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' *- FINAL EVALUATION REPORT - 

1 . INTRODUCTION * * 

The Final Evaluation Report provides an assessment and discussion 

i 

of the activities of the Research for Bettet Schools Career Education 
Program during FY' 197^ • This program is one of-the four employer-based 
career education programs currently funded by the Nat ional "I nsti tute 
for Education. This final report summarizes much Information which has 
bp-^n plahnrgted before (see evaluation report listing in Appendix A), 



as well as presenting new data and analyses principally of a summative 
nature. 

This report is divided into five major'sections . The Introduction 
(Section 1*) prbvides an overview of the evaluation activities and the 
conduct of the project.' Student Population (Section II) describes 
the experimental and comparison groups utilized in*£he project. Forma- 

. i - 

tive Evaluation (Section III) describes the program elements and « 
'how they functioned during FY 197**. Summative Evaluation (Section l«V) 
focuses on the testing of hypotheses posed for the project. Summary 
and Recommendations (Section V) contains an overview of the major find- 
ings and the implications and recommendations of these findings for t 
further program developr^ent and expansion. In addition to the sections 
of text, the following appendices are provided for reference: A. Listing 
of Evaluation Reports, B. Instruments and Testing Procedures, C. Instru- 
ments Developed During FY 197^, D. Procedural Audit Report. 

Research for Better Schools, Inc. (RBS) Is responsible for develop- 
ing, operating and testing 3 prototype of experience-based career 



education (EBCE) . 'This program has been operational I zed In Phi 1 a- , f 

o 

delphta as the. Academy for Carfeer Education (ACE). The Academy 
operated during FY 1 97^ as a licensed private academic school wKh 
senior students receiving their total educational program from the 
Academy. All seniors were In the program for their second year. All 1 
juniors and sophomores were in the program for their first year * 
during FY 197**. Their curriculum differed somewhat as a result of an 
Increasing integration of the>Academy with the Philadelphia Public 
Schools. Juniors and sophomores participated In the "core 11 Academy 



program^, while taking some courses * (foreign Tanquages , cfnwr ii din rut} 
physical^ education, etc.) at their sending school. Students thus 
fell into two distinct groups: seniors who attended the Academy 
courses jonfy, and juniors and sophomores who attended some public 



school, cjourses In addition to Academy offerings. All students par- 



ticipated in the "core instructional component 11 which consisted of 

/ 

three njajor subdivisions: - Career Development, Career Guidance and 
Basic Skills. This core constituted the bulk of each student's 

* i 

academic program with at least \k hours of instruction per week. It 

/ 

Is th/is core program which forms the substance of the present* report. 

The />ther instructional component, Supplementary Programs, was 

• / 
t 

\ available only to seniors and has been discontinued as movement is *~ 

I 

ma<^e toward having the public schools adopt an experience-based 

■ 

career education program. 

J The first subdivision within the instructional core was 

Career Development. This consisted of Career Exploration and Career 

SpecJal Ization activities for students. In Career Exploration - 



students experienced group, programs, (selected by them out of a 
Urge number of possibilities) at commercial and industrial sites 
by employees of the varfoi^ businesses involved. Students parti- 
cipated in these first-hand activities in order to learn about the 
economic community, to test their own vocational interests, and to 
obtain information for their career planning. In Career Special i za- 
tibn students selected a specific career area and Investigated it 

,in depth. These experiences w£re highly indi vidu^T I zed and 
required extensive Interaction at a single community site. 

The~second subtftTMorr-wft-h frh~the-Jjisln^ 
Career Guidance. This consisted principally of small group guidance 
sessions which met each weej<. These sessions focused on life 

^skills', academic motivation, integration of Academy activities, self 

exploration and career planning. Career Guidance also included 

individual counseling of studeats. 1 

The third subdivision within the instructional core was Basic 

Skills. Students were scheduled for activities in an Individualized 
%>• 

Learning Center several times each week. These activities focused 

if. 

on development In Communication Skills and Mathematics and utilized . 
a variety of individual tzed instructional resources. The Individual 
Learning for Adults approach was the primary learning system., 

' These subdivisions combined to form a core of Instructional 
activities which were characteristically individualized and respons- 
ive to student needs. The Academy program was designed to maximize 
student' development and choice within an operational structure 
which could serye a large number of students at feasible costs. 



Tjje evaluation of the Academy program during FY 197** was 
conducted by an internal pnoject .evaluation staff according to the 
evaluation des ign ; spec i f ied 'In the FY ,1 97^ Operating Plans. 'This 
design consisted of berth formative and summative elements. 

The program components most relevant to student instruction 

* 

formed the principal ^subject matter of the formative evaluation 

# ^ 

design. They were: Employer Support, Employer Ut i 1 i zat !on,\Bas i c 
Skills, Career Development, Guidance and Instructional Systems. For 
each of. these ^instructional components, the formative evaluation 
process was organized to address the specific issues identified in 
TR^€va+tta-t4oruplaD and generally to provide the following informa- 
tion: — — _ ^ 

* 

K An explication of the purpose, composition, organization, 
procedures, and operational strategies of the project. 

2. Evidence regarding the degree to which components are • . 
meeting stated objectives. * 

3. Detailed information on the costs associated wi th ,eech 
component. 

This information was gathered by procedures discussed below. 1 
general, the formative evaluation depended upon operational records 
systems, questionnaires and interviews. j 

The formative effort resulted, in the fol lowing /reports : 
REPORT ' DATE COMPLETED 

1.. 15A1 Instructional Systems Design ^ 12/31/73 

2. lOCla Employer Support Evaluation/ 

11C1 Employer Utilization Evaluation 7/1/7** 

3. l2dCl Guidance Evaluation 1 5/15/7** 
%% k. HC1 Basic Skills Evaluation 5/15/7* 

H 



REPORT " .. ' DA?E "COMPLETED 

'.' 5. 15B1 VnSt fuctional Systems Field TSst .V15/7 2 *- 

6. l'lCl Employer Utilization Eval yatjon/ 

Q 13C1 Career Development Evaluation ; 6/ l_/7i» 

• •• ' 

7. 12fC2 Guidance Evaluation II 7/1 5/7^ 



8. 15C1 Instructional Systems Evaluation - 7/15/74 

- . 7 

For each Instructional unit except Basic , Skill j-twa evaluation . 

" # •> 
reports were completed during the year. For Instructional Systems 

, $» 

there were, two additional reports because the evaluation staff 

designed and field tested, as well as evaluated,, that unit. All 

other components were designed and field tested by the developmental 

and operational. staff in each area. This outline of the objectives 

and ^roduc^s^^ U expanded in the Formative v 

Evaluation Section below. s — - — — _ 

Of the many available definitional .differences between summative 

and formative evaluation, none seems to be both ciearly understandable 

and technically precise. .Therefore, a distinction will be" made here. 

based upon differences that experience suggests. For the purpose of 

this report any evaluation activity which j^s related to a specific com 

ponent within the program is considered formative. Any evaluation $ 

activity which concerns the project as a whole or several units in a 

summary fashion is considered summative. For example, a report on the 

Career Development- Uni t employing original analyses of data related 

to that unit, and confined to that unit, would, be formative. .Another 

* * < 

report which reviewed previous analyses and documents related to the 



Career, development Unit and di scussed v £jiem 4 i n the^fibntext of. other 
program units woul d^e* summati ve. .Formative evaluation intends to 
infprm the prpject Staff of un i t -s trengths and weaknesses.. Formative 
evaluattbn also suggests hypotheses to be tested and problem areas to * 
6e assessed in summative evaluation. SumrAattVe evaluation *i ntends to 
judge project conduct and effectiveness, and to present resu-l ts for 
,e xie rna 1 rev i ew . , , 

< * * <»• * 

In^ its design the summative evaluation for FY 1 97^ included a sum- 

mary review of the project support and management components. Those 

reports were considered as secondary in importance, and in some instances 

-they served more properly as documentation rather than evaluation. * 

,The surgmative foqus was given to the analysis' of student effects. It 

wag assume^ that siJch effects ^oul d be a result of the instructional ^ 

components or the program as a *whol e; ' instruct iona 1 components were * 

not treated individually because therewere.no grounds" for hypothesizing 

mutually exclusive effects. Analyses of costs and marketability were 

alscr under taken in the summative effort. Gathering and analyzing employe 

and* parent perceptions of the prograriTTMcewis^were summative .concerns . 

The final area of summative inquiry established was the Institutional^-— 

structure n^£es$ary to conduct a program of thi s type. 

Relating to the above ou&Hne of summative concerns the following 

princlpaj hypotheses were presented for testing during FY* 1.9 7** * 

' - v' •• ■ ■ ■- 

Student Effects 

* • * • • „* 

. 1. Students will gain s itjni f icantly,' (p<. 10) in basic skills over 
the ieourse of the year/ 

. \ 
2. Students will gain significantly (p< . 1 0) more in basic skills 
% than^comparable students in a traditional school. 
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3. Stude^fc^wMI gain significantly (p< • 1 0) in career maturity. 

k. Students, wi 1 1 gain significantly (p<.10} more in career 
maturity tfhart comparable students in^ traditional schoo-l . 

5. -Students will evidence 'a' s Tgnifitantly (p< • 1 0) more positive 
attitude toward school* than students 1 in a traditional school, 

6. ' Students will gain'signif icantly (p< • 1 0) in career knowledge 

over the course of .cluster- experiences. 



Other Effects ' . > 

r « 

\ , Employers wi 1-1 be able to provide learning experiences sufficient 
to meet student needs and interests, 

2. .Employers will evidence a positive attitude and commitment.* ^ 
• regarding the^-progr^m. 

v • . • 

3. Parents will evidence a positive attitude and commitment ? \ 
regarding the program. . . A - * ' * 

4/ Institutional structures will be-establ i shed to enable the 
tho conduct of the program. 

: -~ • y r * 

5.. It will be den>Qnstra-red v that the program can be operated oh 
on -a feasible eost basis. . % 

6. It will b£ demonstrated that there is a ready market for* the 
program. * 

The studenft effects hypotheses were tested using experimental and 
comparison groups with instruments as discussed below in procedural ^ 
sections.. The Resign was a,quas i-experimental one with non- equivalent 

control groups, The testing was done erf"a pre-post basis.- The' "othter 

• » ' . 

effects" hypotheses were not tested statistically. Rather, conclusions 

* *? 

were-drawn from field test,data. .Detailed results were discussed in . 
the jfOl lowing summative evaluation reports:^ ^ , • 
REPORT * ' * 'I DATE COMPLETED" 

1. Report on Management Systems * 1/2/74 ~ * 
Components, - * 

2. * Report on Support Systems Components . 2/28/7V 

0 * 



B REPORT A ^ DATE COMPLETED , 

3. Interim Evaluation Report 3/15/74 . 

A.. B.ook of Measures 7/30/7 4 4 

5. Repqrt on Cost and 'MarketaBi lj ty 9/30/74 

6. Final Evaluation Report 9/30/74. 

Thjs outline of the objectives and products of thesummative evaluation 
is expanded in the Summative Evaluation section .bel ow. 

The evaluation activities undertaken during the course of FY 1974 
thus focused on the formative and summative efforts. In the formative t 

evaluation each instructional component of the project was analyzed 

v 

to provide 'informatipn useful, in* program development and project , 
management. Inspection of operational records, formative questionnaires 
and i fltervi ews * provf ded the appropriate data. The summative> evaluation 
was designed principally to determine the overall effects of the 
program on students, employers and parents.. It also addressed insti- 
tutional feasibility and^planning issues from a program-wide perspective. 
Hypotheses were .formulated and tested using experimental and , comparison 
students, a-pretest-posttest package,, questionnai res and other less 
formal means of measurement. The formative and summative efforts 
exhibit many interrelationships. 1 Formatjve, analysis indicates the 
degree to-which the p^&gram design was actually implemented. This 
suggests the level" of summative effects to be anticipated and provides 
a context for thei r "interpretation. Formative r^siil ts al^o may suggest^ i 
hypotheses to be tested In a sumijiative design. Summative results may « 
indicate areas of curriculum and procedure which need further deVelop- 
ment^n subsequent project years. ,/,' 



Both the formative and summative efforts during FY 1 97^ resulted 
In extensive reports as indicated above. In addition to these, a 
series of Special Reports was issued* 

REPORT < DATE * COMPLETED 

]. The Stu<&nt Recruitment and 1/30/7 7 * 
Selection Process <?* 

2- Report on Student Characteristics 2/28/7^ > 

: 3. Report to Employer-Coordinators \^ e 
on the Results of Si*th Quarter , \ \ 



Interviews \ 



jh. Report tm Instructional Units for \ hf\n^ 
Fifth Quartet 

Activities outside of the formative and summative plans were 
also undertaken. Principal among these was an extensive, instrument 
development effort. Four instruments resuj ted from a. cooperati ve 
'process involving N1E and the evaluation staffs of the four EBCE 

projects: ^ 

' J. Student Demographic Data Questionnaire ^ 

2v Student Opinion Survey * ; - 

> 3. ^Parent Opinion Survey ( 
h. . Experience Site Demographic Data Questionnaire 
These instruments were analyzed and -f i'na 1 Ized by the RBS Career Educa- 
tion Project evaluation staff for use by all four projects^ 

Two additional instruments we-.re. developed exclusively by RBS: 

1. Cluster Tests of Knowledge 

i 

2. Assessment of Student Attitudes Sc^le 



in 



Each of these instruments along with associated analyses has 
been include4 v in Appendix C of this report. In most cases the 
results '.obtained indicate that further development is required for 
these measures, but they functioned fairly well and are presently 
acceptable for use. This instrumentation effort was considerable in 
light of the measurement obstacles inherent in experimental programs, 
and it represents a substantial advance in Career Education Program 
assessment . 

The instrumentation process occurred within the context of a 

i 

broader cooperative research activity established by N I E among the 
evaluation staffs of the four projects. Evaluation directors and 
staff members visited each of the four project sites toexchange 
information on the projects and confer on various evaluation issues 
related to the Career Educat ion^rogram. Joint conferences were also 
held at the annual meeting of the American Educational Research 
Association (where a symposium was also presented) and Princeton 
University. Continuing relationships were thus established among the 
individual projects and NIE. This resulted in the sharing of exper- 
ience and numerous cooperative ventures. 

Extensive resources were also devoted at RBS to computer! zed 
data systems development. It was attempted to design and implement 
automated records systems capable of managing operational tasks ( o .<). , 
attendance, grades and transcripts) as well as maintaining a companion 

slve data file for evaluation and research purposes. These systems 

i 

were developed to varying stages of completeness* In all cases the 
point of field testing was reached, but the systems were in need of 
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further development before they could be operated by Afield personnel 
withouta need for technical assistance. * 

In summary the evaluation activities for FY 197^ were in several 
major areas: formative evaluation, summative evaluation, instrument 
development, cooperative research and data systems development. The 
outcomes of these activities are presented in the remainder of this 
report.; 

-I I . STUDENT POPULATION 

The purpose of this section of the final evaluation report is 

to present summary data on students whoparticipated in the Academy 

* ^ 
for Career Educatl ori~(ACE) program In FY 7** and students who were 

selected for comparison groups. The students in the Academy program 

comprised two groups: 

1. ACE Group - (n = 76) These were 12th grade students who 
received their full high school .experience at the Academy. 
They have been in the program since September 1972. 

2. ACE-Olney* - (n - 76) These were 10th and 11th grade students 
who" participated only.* in the core Academy, program: Career 
Development ^--Wreer /Guidance and Basic Skills. They have 
their remaining courses at Olney High School, participate 

in Olney's extracurricular program, and will receive their 
diploma from Olney rather than the Academy. They started the 
program in September 1973. 

Two additional groups of students were selected for the purpose of 

comparing tfie progress of Academy students with traditional program 

students. * These comparison groups were selected from the Olney High 

School student body: 

> 

3t Comparison Group - (n = 28) These students applied for the' 
Academy program, were accepted, but eventually declined to 
I j enroll. They were selected for comparison because they 
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volunteered for a career-oriented program (evidencing 
a level of Interest), and they passed the program's 
requirements* They were all 11th graders. 

*K Context Group'- (n = 81) Thele student represented a 
random selection of equal numbCTS of 10th, 11th and 12th 
grade students from Olney High School. They were selected 
to provide comparative data on the "typical" Olney 
student. 

The comparison group students were not involved in the* Academy pro- 
gram, but were tested on the same measures as- the Academy student 



groups. (~ 

The fol loving instruments were included in the pretest-posttest 
evaluation design. All were~adminis tered to both experimental and 



control groups. A detailed discussion of the procedures Is included in 



appendix B of this report. 

1 • Comprehensive Tests of Basic Skills (CTBS) - Th i s i s a 
standardized test of 'basic academic performance. The 
Reading and Arithmetic subtests were used. 

2. Career Maturity Inventory (CHI) - This is a standardized 
test of career attitude and several areas of career-related 
competencies. 

3. Assessment of Student Attitudes 0 Scale (ASA) - This fnstru- 
ment is currently underdevelopment by the eval uation^staf f ♦ 
It is intended to measure attitude toward school and several 

. elements In the learning environment. 

. • 

J+. Student Demographic Data Questlonnai re (SDQ) -*This instrument 
was constructed by the evaluation staffs of al 1 Experience- 
Based Career Education projects to provide common data on 
basic characteristics. 



Background Characteristics 

Table 1 presents several summary characteristics of the student 
groups involved in the FY 1*97^ .program. At the start of the academic 
year each experimental group (/^CE and ACE-Olney') contained 76* students, 
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» TabU 1 

Composition of Student Croup* 



Croups 
Characterise lca^s^ 


ACE 


ACE-Olney 


4 

Comparison 


Context 


Total 


1. Size 


76 


76 


28 


61 


261 


2. Average K%t 


17.0 


15.3 


16.3 


16.3 


16.4 


3. Crnde Level 


12.0 


10.5 


— "i — 
11.0 


11.0 


11.2 


4. Prcvloun School 
X Attendance* 


89.8 


91.6 


90.5 


88.7 


90.1 


5. Previous School 
CPA ** 


3.5 


3.0 


3.3 


3.4 


3.3 



ACE Group = Students originally recruited for FY 73 program, all 
equivalent of 12th graders, all in program for second year. 

ACE-Olney Group = Student recruited for FY 7** program in cooperation 
' with Olney High School, grade equivalent split between 10th and 
11th graders, all in program for first year. 

Comparison Group = Olney students who applied for ACE-Olney program, 
were accepted, but finally decided to not enroll . 

Context Group - A random selection of equal numbers of 10th, 11th 
and 12th graders from Olney, no known exposure to program, no 
intended selection biases.^ ■ & 



k * Data completeness = 8^3, scale 03 - 1003 

* * 
** Data completeness = 963, scale J = high t& 5 = low 
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The context group (randomly selected) was roughly equivalent, with 81 
students, while the other exper imeinta 1 grouf) (the comparison students 
who had opted out of the program) was much smaller with only 28 
students. The total number of subjects available for analysis was 261. 

Age , In age, the ACE group was approximately one year older, on 
the average, than the other groups. Likewise in grade level, the 
ACE group was approximately one level higher th.n the others. The 
ACE group consisted entirely of 12th graders. The ACE-Olney group was 
about evenly divided between 10th and 11th graders. The Comparison 
group consisted entirely of 11th graders. The Context group was 
divided among the three levels. 

Previous School Attendance . Reported previous School attendance 
was uniform across groups and high (90%). t It seems likely that this 
figure has been subjected to error somewhere along the way, but it 
reflects school records as accurately as they could be analyzed. 

Previous School GPA . Previous school grade point average (GPA) 

varied slightly across groups. On a 5 point scale with 5 being low, 

the ACE group averaged 3-5 (C-),'the ACE-Olney group averaged 3-0 

(C) and the controls fell roughly in between. This would indicate that 

i 

the 12th grade experimental s have a relatively poorer past school 
record while the 10th and 11th grade exper imer.ta 1 s have a relatively 
better one. Relationships to other measures and present performance 
rShiain to be dVawn. 

18 



Race and Sex . Percentage calculations on each student group are 
presented in Table 2. The sex distribution was roughly equitable 
except for the Comparison group, where females exceeded males almost 
2 to 1 . Racial distributions were more varied. The ACE-Olney 
group was preponderantly Black (81%), while the others evidenced 
close to a 60%-k0% split. Sex by race breakdowns showed that the 
typical size ordering was Black Female, Black Male, White Female ■ 

* 

and White Male. The Comparison group had an unusual distributions 
indicate that the program is differentially attracting population 
subgroups. The reasons for this bear investigation. The dis- 
proportionalities also have implications for analyses to be presented 
in the Summative Evaluation Section. » 



T«bU 2 

Sex end fUce Distributions 
In Percentt 

6 



Group 












ACE 


ACE-Olney 


Comparison 


Context 


Characteristics 










Hale 
Female 


*9 
SI 


56 


35 
65 


53 
<»7 


Hack 
White 


67 
33 


81 
19 


61 

39 


53 


Black Male 
Black Female 
White Male, 
White Female 


37 
19 
U 


til 
M 
15 
3 


17 
kk 
17 
22 


27 
27 
27 
19 
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Parental Occupations . Tables 3 and *t present the distributions 
of parental occupations for the various groups. No major systematic 
differences between groups v/ere observed. Preponderant" ca tegories 
for fathers were: Operative - 33%, Craftsman - 16%, and Laborer - ! 13%* 
The largest categories for mothers were: Housewife - ^2%, Clerical 
18%, Operative - 13%, and Services'- 12%. 

Parental Educational Levels . Tables 5 and 6 present the distri- 
butions of parental educational levels for the ACE and ACE-Olney 

° • ! 

groups. Mo$t parents were reported to have completed some or^al 1 of 
secondary school. Since the levels have some scalar quality, an 
informal {test for differences between the groups was made. The 
average educational level of the ACE group was 3-91 on a scale of 1 
to 8 with 1 low and 8 high; the ACE-Olney average was 3.97. The 

groups thus seem similar with regard to parental occupation. 

v - ■ — . 

Post Secondary Plans . All groups were questioned about their 
post secondary plans. Table 7 presents this information. The two 
experimental groups did not seem to differ markedly with approximately 
25% planning to immediately enter a vocation, ov.er 50% planning 
further education, and about 10% anticipating job training. The 
ACE-Olney group was relatively more interested in *t year colleges . 
within "further education" by a margin' of 35% to 22%. Of note is the 

» 

wide disparity between the experimental and control groups. In the 
case of the latter, for Comparison and Context groups respectively, 



fully 50% and 40% were planning immediate entry into a job, **2% and , 



48% were planning further education, and very few were anticipating 

•?.o 
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Foit SacgcuUry Plana 
in Perc*nti 





^^^^^ Crouf 
Catagary ^*>*v^ 


AOt 


ACt-Olftay 


Caooarloon 


Contost | 


♦Total 


• 




1, Eoolayoont 


If. * 


17,4 


jU.G 


\a J H 

1 


29i9k« 






2, Jao Traiatag 


1<M 


0 

••1 


4.2 


v. 4 


■ ' i 
o. I 






3. Military 


* 

7.} 


4.3 


A A 
V.W 


i 4 1 
1 


3/4 




» 




0*0 


0.0 > 

" / 


y o. 


4 0.0 
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anything else. From these distributions It seems that, while al! 

groups have a high Incidence of planned post secondary education, 

the remaining control students aire more Interested In Immediate employ 
■ment than the remaining experimental subjects. The plans of ACE and 

ACE-Olney students were more evenly spread over .the categories 
^va^lable (possibly "indicating more diversity of interest); the Com- ■ 

partson and Context>dlstrlbutlons were blmodal within the employment 
„ and further education categories. - \ 
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Reasons f or Academy Enrollment * Regarding reasons for Academy • 
enrollment (Table 6), the opportunity for Career Exploration was the 
largest single factor for both ACE*and ACE-Olney students, Indlvi- 
dualized Instruction, choice of courses, smaller classes and 

.the opportunity to move around the'clty were also Indicated by a sub- 

t 

stantial number of students. 
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In Investigating the Background Characteristics of Academy stu- 
dents and, their contrpl group counterparts descr 



|cs have 

been presented above. The students Involved seeta to be representa- 



tstfes of Academy 
:rlptlve statistic 



tlve of an urban population. Between group differences were fount! 
on the following dimensions: race and sex composition, preVfc^s 
school GP/\, and post secondary plans/ ThfTftCE group (12th graders) 
was approximately $0% - 503; in sex composition and 70* Black, but . 
' the ACE^Ine^group (10th and 11th graders) was approximate! y\5S% 
Male and 80*'B1ack. I<n the ACE-Olney group only 2 students were* 
White Female and only 10 were White Male. This disparity was not 
Intended, and* the reasons for its/ occurrence should be Investigated. 
I^f the same phenomenon were to occur next year (after the iZt* graders 
had graduated),, the net result would be a program. th^t served only 
one race* of students. ACE students had, a lower grade point average 

4ft — * 

fromHhefr sending schools tham-did the ACE-Olney , students (C- vs. C) * 
The control group students averaged In between the experimental groups. 
'The control group students were 1 more interested In Immediate post; secon 
daVy employment than were the experimental students. The experimental 
students had more diversity In their planning. All groups had a high 
level jof interest -In post secondary education. In summary, the ACE- 
Olney group wa>s In compos It ton more Male and Black than the ACE group; 
It also had a better achievement ^record. Sex and racial data Indicate 
that the Comparison* Group Is not a comparable control grojup. Both the 
ACE and ACE-Olney groups had more dfters^ty of post secondary inl<* tests 

than the Comparison and Context groups; they also expressed less intere 
"I ' 

'In ^Immediate employment aftfer high school. , 



Standardized Tests • 

; f • * T 

The pretest results of the standardized tests used tn'the evalua- 

tlon designee presented below. Posttest ^fnfrlngs are presented In \ 



*. the Summative £ valuation section of this report. • \ 

* / 

Career Maturtty Inventory . Tables 9 through 13 present the 
resultS'of the Career Maturity Inventory pretests. InalLc^es 
percentile; were used fpr analysis purposes; thus grade level dlf* 

ferenqes have been taken* into account. Each subtest table consists 

0 ♦ 

j .of three subd Ms-Ions. The first presents basic descriptive data 
ok each group; the' groups are arranged In order of the reign I tude of, 
x their mean score* The. second is an analysis of variance* table wh&ft* 5 
Indicates the degree of statistical reliability with which the largest 
.mean c^n Be considered different from the smallest mean. The third \\ 
section indicates the statistical significance of the remjilnin^mean 
differences, hi all cases th$ 'numbering of means reflects the order 

;. - ^ 

„ printed 'In {he first section, of the table. 

Inspection of these results allows some interesting collusions: 

1. In no case was the Comparison Group different from,thp Ebntext 
Group (control groups) ^ * , 

2. In no case was the ACE Group 'different front the. AC^rOlney 
Group (experimental groups) 4 

, • :.' » • 

3- In most cases both exper knental groups were superior to 
both control groups 1 >p > ' * 

"tyese findings indicated no differences' between first year (AC Ej • 

-and second year (ACE-Olney) students qn career maturity" factors as 

measured by the CMI. These findings also showed hb differences between, 

students who applied, were accepted, but dropped, from thfe program 

(Comparison^ and a random Ulecijon of students (Context). Howey^r, 

. * * • x< 

tioth of these groups were "consistently lower than the experimental 

' * * *• 

t groups. "This suggests that at least one consistent difference between 

students who stay In the program and other students their age Is a - " 
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higher evidenced career maturity. Thts must have been a factor at 
work In the recruitment and selection process, and merits further 
study.' In percentile categories the control subjects were generally 
close to the lowest quartlle while experimental subjects were close 
to the middle; 

- — " <• 

I 

' > * 

Comprehensive Tests of Basic Skills. Tables \k through 20 pre- 
sent the results from the Comprehensive Tests of Basic Skills In the 

0 ( 

same format* used abovt for the Career Maturity Inventory, The Reading 

* 

and Arithmetic subtests were 'administered. Since scale scores were 

** * * 

available for this Instrument, they were used for analysts purposes to 
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Increase discrimination and accuracy. Grade equivalents are reported 
- In the text for descriptive purposes. Since scale scores do not 
equate for age, the ACE group's pretest scores for last year were 
employed In this analysis. 

The statistical procedures demonstrated no significant differences 
between groups on the Arithmetic subtests. However, on each of the 
Reading subtests the ACE group was significantly superior to the Com- 
parison group. The Implications of *hls single comparative difference 

« j 

are not easy to draw, but It seems clear that the ACE students upon 

entrance to 11th grade were better In reading skills than the students 

J 

who applied but dropped one year later. Jhe lack of other significant 
differences would suggest that any other group comparisons relating to 
basic skills could be made assuming an initial equality of group 



achievement . 



«(• 



/ . 



Grade equivalent averages for eacb^group are presented In Table 
21. As can be seen, most groups were functioning at the 7th or 8th 
grade level on the average. Arithmetic scores were generally 
lower than Reading scores. All groups ranged on all measures from 
a low of the 3rd. grade level to a high of the 12th grade level. 
From these low scores It is apparent that basic skflls Improvement 
Is a priority need, for these students. 
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Assessment of Student Attitudes Survey , The Assessment of Student 
Attitudes Scale rs not properly a standardized instrument* but rather 
a measure which has been given much developmental attention by the 
evaluation staff. It Is included In this section because It Is a part 

c 

V 



of the pretest-posttest package aimed primarily at summative evalua- 
tion. 

The Assessment of Student* Attitudes Scale was designed 

f 

specifically to measure student attitudinal dimensions in several 
areas central to the instructional process. These areas have been 
designated as subtests with separate scores as follows: 

1. Education in General 

2. Program Curriculum «}> 

3. School Facility 
i*. Program Counseling 

m 

The Instrument is intended to measure student attitude toward the 
school program with reference to each of theareas listed above; The 
instrument is generalized in content and phraseology* t<^be applicable 
In public schools as well as alternative or experimental programs. 
Extensive rel labll ity,, val idlty and discrimination value studies have 
been conducted; the preliminary results of these studies are included 
In appendix C of this report. ^ 

Tables 22 through 26 present the results from pretest analyses 
of each ASA subtest. The tables have been formatted in the same 
manner as those reported above for the Career Maturity Inventory / 
and the Comprehensive Tests of Basic Skills. Data are reported as 
raw scores reduced to a mean on a scale from 1 (low) to 5 (high). As 
can be seen from these results only one difference between groups was 
found to be significant; that was between the ACE and ACE-Olney groups 
on the' Attitude Toward Program Resources Subtest. All groups were found 
to be not different on all other subtest measures. It is apparent that 
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all student groups are similar with regard tb attitude toward school 
as measured by the RBS-designed Assessment of Student Attitude Scalp. 

I v 

' The following summarizes between group djfferencfes found in the 
analysis reported above: ^ + 

1. CTBS Rpading • ACE > - Comparison - 

2. CMI (most subtests) ACE ■ ACE-Olney > Comparison - Context 

3. ASA (subtest 3) ACE > ACE-Olney 

In all other comparisons the groups were found to-be statistically not 
different/ 

Experimental groups (ACE and 4 ACE-Olney) were superior to control 
groups (Comparison and Context) on most subtests of the Career Maturity 
Inventory with average percentiles of approximately kQ vs. 25* On the 



Comprehensive Tests of Basic Skills all* groups v*re.$rtatistlcal ly 
similar, except that the ACE student group was- superior .to*. the ' 

r - 9 C . > ' 

Comparison student* gro t up. Student scores ranged from the 3rd grade 

* * t 

level to the pofct 12th grade level with post averages around the 7th 

or 8th*ghade ,equlvajency. The Assessment of Student Attitude Scale 

results indicated that all groups were similar, except that the ACE 

group was superior to the ACE-Olney group on the Attitude Toward Pro- 

gram Resources subscale. v 

Summary 

Tills section of the final report has presented information 
on the characteristics of students involved4n the RBS Career Education 
Program during FY 197*»- There were two experimental groups enroled 
in the Academy for Career Education: the ACE group of second year 
students' and the ACE-Olney .gfoup of first year students. There were * 
also two comparison groups who participated in a traditional school 
program but were 'given the same tests as experimental students within 
the evaluation design. These groups were a Comparison group of students 
who had applied to the program and a Context group of randomly selected 
high school students. 

All groups were given a battery of fou^pretest Instruments: 
jthe Student Demographic Data Questionnaire (SDQ), the Comprehensive 
Tests of Basic Skills (CTBS), the Career Maturity Inventory (CMI), - 
and the Assessment of Student Attitude Scale 'ASA) . 

The SDQ Indlcajted th»t the four groups were fairly similar In 
constitution as Judged background characteristics. It was found 
that experimental groups tended to'have more diverse post-secondary 



plans than coritrbl groups- Some differences In past school perforipance - t 
we're evident. It was also'apparent^thaT^over the first two years of 
progVam operations a trend toward a Black student population has 
been In effect. 

6 

The CMI revealed that experimental groups Were generally-superior 
at entry to control groups on career maturity factors as measured t>y 
this Instfument. This presents a consideration for subsequent analyses 

c • 0 $ 

and suggests the incurrence of a selection bias. 

\ . 

The CTBS results did not Indicate consistent differences between 
groups at the pretest time. The ASA likewise did not support any 
consistent differences between experimental and control groups. . 

The findings presented In this section concern only the , 

\ • i 

pretest administration of evaluation Instruments. These results are. 
Intended for descriptive use In viewing the samples Involved In the 
study. For posttest results and application Of analyses Intended to 
depict program effects see the Summatfve Evaluation Sectjon below.-. 

• \ . < 

» 111^ FORMATIVE EVALUATION e 

.. • \ ^ . . . ; 

Introduction V 

The formative eva\^tlon was designed to gather information useful 
for program development and ^project management. Only the components 
directly related to the provision of I nstructi ^rv s were^ included in 
the formative evaluation design; they^were the Career Development Unit, 
.the tfareer Guidance Unit and the Basic Skills Unit. Issues dlUussed 
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in each of the evaluation unit's task reports will be summarized 
in this section and new data will be introduced where it is available. 
In addition to these issues, a final section which endeavors to discuss 
how the formative evaluation was used in program redirection and redesign 
will be included. 

Career Development Unit ^ 

Overview . The following tasks were performed in the evaluation 
of the Career Development Unit in FY 1 97^ - 

L Evaluate the employer contact process; 

2.. Assess the nature and extent of employer involvement;; 

3. Evaluate employers 1 curriculum; 

*\ 

* k. Evaluate integration of employers 1 activities within clusters; 

5. Determine adequacy of potential employer pool in relation 
to student needs; 



6. Identify student needs and interests;/ 



7. Measure levels of student knowledge In relation to goals 
and objectives; and- t / 

/ 

8. Survey student attitudes regarding offerings. * 



Tasks were documented In Task Report lOtl/HCl; tasks 5~o were 

/ 

documented in Task Report 10Clb/l 3C1 . Following are the summarized 
findings of each of the evaluation tastes. 

. c. 
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Task 1 - Evaluate Employer Contact Process . The employer con- 
tact process utilized for soliciting new employers into the Academy 
program was found to be effective in obtaining an employer pool for 

career explorations and spec! a It zM ions . The Greater Philadelphia 

* * * ' 

6A ■ 



Chamber of Commerce was given the major responsibility for contacting 

employers and serving as a liaison between the economic sect<?r and 

the Career Education Project. 

Procedures developed f6r the identification and recruitment of 

employers, for participation- in the Career Exploration program fell into 

three phases. Phase one consisted of the actual identification and 

recruitment of potentially involved employers. Phase two involved the 

• ** 
securing of a commitment to develop a program in Career Education. 

Phase three consisted of the operational ization of tl\4 Employer Program 

Development Plan. Detailed descriptions of each of * these phases and 

\ s 

of this process may be found in Task Reports 10A1 and lOtlb, 

4 t 

Sixty-seven (67) employers participated in the Career Exploratfon 
program in FY 1974. Of these, 40 or 60$ were newly recruited. These 
67 employers were used to provide 14 exploration clusters in the Fifth 
and Seventh Quarters and 16 in the Sixth Quarter. For a complete list 
of these employers see T*sk Report lOCla/HCl. 

The procedures developed for the identification, contacting and 
recruitment of employers for participation in the Career Specialization 
program were-sfmilar to those used in the Career Exploration program 
except that the student had the prime responsibility, in concert with 
his counselor-coordinator, in contracting for his learning expedience. 
A total 6f 40 career specializations were provided by 23 employers dur- 
ing FY 1974. A list of these employers and specializations is presented 
in Task Report 10qia/llCl. 
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* 4 

Task 2 - Assess the Nature frnd Extent of Employer Involvement . 
The Career Development programs were Implemented in employer, union 

and agency settings; all three were considered a* employers. In FY 

t 

\S7k a total of 84 employers (an increase of 83% over FY 1973) 

i 

participated In the Career Development program. Following is a 
breakdown of that particpation in both years by instructional component, 

Instruct Tonal Component FY 1973 FY 1 97 1 * 

Career Exploration Only 31 60 

Career Specialization Only 7 1** 



Career Exploration and 
Career Specialization 



8 10 
TOTAL: "15""' TPT 



In addition to this instructional involvement, employers con- 
stituted $5% (11 of 20 seats) of the total composition of the Board 
of Directors of the Academy for Career Education, the pol i cy r sett i ng 
Qrgan of the Career Education program. A further level of employer 
involvement included their recruitment of employers and students for 
participation in the Career Education Program. 

\ 

Tas k 3 - Evaluate Employer's Curriculum . The curricul uhi of the 
Career Exploration program consisted of the instructional materials and 
strategies implemented in employer courses. The fact that less than 
half of the Career Exploration courses produced Employer Course Plans 
weakened the evaluation effort considerably. In addition, where* 
Employer Course plans were extant, they were in most cases not based 
upon si te analyses. 

The Employer Course Plans, whi ch "were submitted for evaluation 
were found to be well sequenced and well related to the learning objec- 



\ 
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tlves of the course where those object Ives were specified.. However, 
the specificity with which those objectives were stated was found 
to be low. 

The curriculum for the Career Specialization program consisted of 
the experiences 'specif ied in the individual contracts. Within the 
context of the goals and objectives of the Career Specialization pro- 
gram, each participating student and an employer representative 
developed a program that met with their mutual approval. Each con- 
tract was to describe the specific learning" experiences agreed upon 
by the student and the participating employers 1 and approved by the 
Counselor-Coordinator. The contract was to include a description of 
(1) instructional objectives, (2) instructional activities, (3) a 
management plan specifying times and locations for the instructional 
activities, (4) a. means of evaluating the student's learning, and (5) 
the responsibilities of all parties to the aqreement. 

Contracts existed for only 23 of the career specializations. Pos- 
itive ratings by \he evaluator of the contracts as curriculum descrip- 
tors were the relatbdhess of learning activities to objectives and 
the inclusion of statements of objectives and learning activities. v — ^ 
Negative ratings Included the specificity of the statement of objec- 
tives and the lack of a specified sequencing of learning activities. 
. Identical contracts were noted for many career specializations for 
which' there was multiple participation. It was questioned whether this 
duplication was compatible with the program coal of Individualization 
of programs to meet each student's needs and^ whether students were actually 
participating In the contracting process. 
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Task jrj; Evaluate Integration of Employers 1 Activities Within 
CI usters» Although cluster meetings were held to aid in the review 
of program objectives, the establishment of cluster objectives and 
the implementation and planning of programs, these efforts were seen 
as having been minimally effective by Employer-Coordinators and 
Counse lor** Coord i na tors ♦ 

Task 5 Z Determine Adequacy of Potential Employer Pool in 
Relation to Student Needs . Two criteria were used to evaluate the 
adequacy of the employer pool in relation to the student population 
-of the Academy for Career Education, student needs and student 
interests. 

In the Career Exploration program student needs were determined 
by the program requirements of 6 career explorations and prior 
paVticlpat ion* To meet student heeds, the employer pooV had to accommo- 
date 125 students in the Fifth Quarter, 92 students in the Sixth 
Quarter, and 84 students in the Seventh Quarter. No Career Explorations 
were offeree! In the Eightho Quarter (Summer session). The student 
capacity of the Career Exploration p'rogram was 1^0, 172, and 152 stu- 
dents in the Fifth, Sixth and Seventh Quarters respectively. 

Student interests in the Career Exploration program were deter- 
mined prior to registration and recorded on the Student Needs and 
Interest .Form. These forms were available only for twelfth grade stu- 
dents in the Sixth Quarter and for all but newly enrolled students in 
the Seventh QuarteV. When actual enrollments were matched against 
stated preferences for clusters, the ability of the pool to meet stu- 

o 
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dent Interests was 90* ?n the Sixth Quarter and 1 83* In the Seventh 

t 

Quarter. A complete discussion and presentation of data may be found 
In Task Report 10Clb/l3CI . 

Since career special izat:ions are optional, student needs were 
defined by student interests. As in the Career Exploration program, 
student interests were elicited through the use of the Student Needs 
and Interests Form. 

For the Sixth Quarter, 35 twelfth grade students expressed an 
'interest in part fcpatlng in career specializations. Among the career 
area In which interest was expressed were auto mechanics, child care 
worker, commercial artist, computers, electrician, insurance, law, 

m 

medical secretary, physical therapy, social work, teacher's aide, and 

c 

travel agent. 

For the Seventh Quarter, 37 twelfth grade students and II tenth 
and eleventh grade students expressed an interest In participating in 
a career spec ial Izat lon^ Amortg the career areas in which interest 
was expressed were artt N bookkeeping, carpentry, electrician, government, 
hospital work, journalism, retailing, tutor, and veterinary mediciye. 

In the Sixth Quarter, 22 career specializations wore provided; 
19 of these were provided to twelfth grade students. lhhty^fiv«« (V,) 
twelfth grade students had expressed an intere.l u\ hiumt sper kiU/.i 
tion; 13 or 37. 1£ were successfully placed in uiver specializa- 
tions. Another 6 twelfth grade students who expressed interest after 
the preregist ration period were al so. successf elly placed In career 
specializations. Of the 35 students requesting career specializations 
in the Sixth Quarter, 22 were not placed. 



0 

In the Seventh Quarter, It tenth and eleventh grade and 37 
twelfth grade students had expressed an Interest in career specializa- 
tion. Of these no tenth and eleventh grade and 10 or 27.2? of 
the twelfth grade students/were placed In career specializations. Thus 
all 1! tenth and eleventh Vade students and 27 twelfth grade students 
expressing interest in career specialization were not placed. 

The employer pool was not yet sufficiently developed to provide 
career specializations in the quantity necessary to meet student needs. 
Career Specializations could be arranged for only 23 of the 83 students 

s 

who expressed interest in such an experience. , While the Counselor- 
Coordinator responsible for this program reported that many of these 
students changed their minds either before or after cpntacts had been 
made with employers, he also stated that in the case of 21 students 
employers could not be obtained. Thus, at best, only 23 of kk or 52.3 
% of the knowo, unchanged requests could be met over the two 
quarter period. 

Task 6 - Identify Student Needs and Interests . Ab determined 
through the administration and analysis of fhe Student Needs and 
Interests Form, the capacity 6f the Career Exploration program necessary 
to meet minimal student needs in FY 1974 was 125 students in the Fifth 
Quarter, 92 students in the Sixth Quarter and Bk students in the Seventh 
Quarter. 

Student interests, as indicated on the above mentioned forms, 
were as fol lows : 
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A full discussion of these results may be found In TasK Report 
10C1b/13C1. 



Task 7 ~ Measure Levels of Student Knowledge In Relation to 
Goals and Objectives . In relation to the Career Exploration courses 
measures of student knowledge met with limited success In FY 1974. 



staffing problem within the evaluation unit mitigated against the 
development of cluster tests of knowledge as had been previously 
planned. "The Career Maturity I nventory* (CM I ) was administered 
and the results are discussed in the summa'tive section or this 
'report. While there were decreases in average grades in Career 
Exploration courses, it was not determined if this reflected 
adversely on student carder knowledge or was attributable to other 

0 

factors (poor attendance, etc.). 

A measure -of student knowledge in Career Specializations was 
the ability of the students to list job skills they had learned 
and academic skills that were needed in the career area. The 
Career Specialization Student Questionnaire was given to the 
22 stud"ents who participated in career specializations in, the Sixth 
rter; 15 students completed the questionnaire for a 68.2 percent 
rate of return. Of these, 12 or 80.03; listed specific job 
i^skills they learned about In their career specializations; 10 students 
or 66.7% listed specific academic skills that were needed \*\ 
the career area. 

The average grades earned for career special i/af ions wort- <ils<» 
inspected. In the Fifth Quarter, the average gradr wn«s slightly 
below a B; In the Sixth Quarter the averaye was rj BK tiMdt l<> 
Indicate the students are learning at or dbovp a B h ut 1 . 

The responses on the Career Specialization Student Questionnaire 
indicated that most" students learned about specific job skills in the 
career are^ of their specialization but that in a third of the cases 
they could^not relate academic skills to proficiencies needed in the 



!• Determine effectiveness of Career Guidance Groups In 
, conveying Career Knowledge; * 

2. Ascertain the composition and utilisation of the Needs. 
Research and Personal Ppsition Audjt; 

3* Survey staff and student attitudes toward Transactional 
• * Activities; 

H. Evaluate student outcomes of Placement Program; and 

5- Analyze case studies of student career development. 

% 

Tasks 1-3 were documented In Task Report l*2dCl ; tasks k and 5 
were documented in Task Report 12fC2. Following are the summarized 
resell ts of eacK of the evaluation tasks. 



Task 1 - Determine Effectiveness of Career Guidance Groups in 
Conveying Career Knowledge . Three direct measures of student career 
knowledge were usee! In FY 197*» the Tenth/Eleventh Gr^de Guidance 

oup Questionnaire, the Twelfth Grade Guidance Group Qqestlonnoi re, 
artd the Career Maturity Inventory* The first two instruments were 
designed by the evaluation staff to measure how effectively the Guidance 
Groups conveyed career knowledge to 'the combined tenth and eleveath 
grade and the twelfth grade students. Separate curricula were 
Resigned for \hese two groups because of differences in inntruct tonal 
emphasis. The Career Naturi ty Inventory K <i 'standard i/«-<i test wliirj) 
was .chosen as a common measure. of career maturity *j»>d knowing- l»y * 
the evaluation staffs; of the Experience- Based Lureer Education iTojetls. 
The combined tenth/eleventh grade and twelfth tjrado Guidance 

V- 

Group Questionnaires werev administered to a 15V* random sample (n * 20, 
10 students in each group) between March 6th and March 13th t 197*. 

. \ 

These instruments were designed to determine what sorts of knowledge 

\ 4 
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the Academy students had been gaining in their Guidance Groups 
relative to the curricula developed by the Career Guidance staff. 
The Guidance Group Questionnaires were developed from, and in con- 
cert with, the object Ives #of the Guidance Group curricula designed by 
the developmental stafffof the Career Guidance Unit. The curriculum 
developed for the twelfth grade students placed the greatest emphasis 



or>— sic Ills such as resume writing, standardized testlnq rationale, qoal • 
setting, and the preparation of employment applications. The objec- 
tlve curriculum developed for the combined tenth/eleventh grade p 
students placed the greatest emphasis on standardized testing 

j 

.rationale, future studies, knowledge of the Academy program* and 
the differences between fact, value and opinion.' 

The tenth/eleventh grade random sample students who re 
administered the Guidance Group' Quest ionnai re (n * 10) exhibited the 
ability to differentiate between values, facts, and opinions. In 
addition, they were able to identify the Academy's computerized career 
information system by its acronym.. The students were not able, on the 
whole, to identify basic operational fadets of the one program 
element (Life Ski lis Specialization^ included In the questionnaire. 
This Material was expressly irfduded in Jhe instructional materials ^ 
for the Sixth Quarter because of the small percentage of students 
who took advantage of the Life Skills Specializations in the Fifth 

Quarter. Questions deal Inq with standardized tests. Life Skills Soeciallz*- 

f 

* * 
tions, and future studies yielded mixed, but qeneral ly poor , results. The' . 

students showed a general understanding of the material but did poorly 

on the more specific Items. 

err? 



The twelfth grade random sample students (n ■ 10) exhibited 
contrasting levels of ability as measured by the -Guidance Group Question- 
naire. On the one hand they were able to identify the correct ■ * 
responses to the multiple choice items dealing with a resume; but 
were on the other hand incapable of correcting the letter of applica- 
tion. While the students were able oto Identify which standardized 
' tests they had taken during the year, they could not identify the 

■ / • 

. correct^ response concerning the rationale for standardized testing. 
* Eighty percent (80%) of the' sample were able to identify the 

acronVm of the computerized career information system* but only 50% 
.of the sample knew how to use the system. This indicated a general 

failure to apply, or adequately comprehend,, the material which was 

emphasized' in the Guidance Groups. 



Task 2 - Ascertain the Composition and Uti lization of the Needs 

Research and Pergonal Position Audit . The* Meeds Research ^and Personal 

Audit was an instructional, guidance oriented Activity which took place 

-* * 

in the tenth/el eveatti grade Guidance 'Groups in the fifth Quarter of 
Academy operations. {.The composition *of the Needs Research and^Per- 
sonal Posititfrj Audit was determined, by inspection of Task Report 
!2bAl and*l t 2bA2, tQ consist of the following instructional units: 

0 

1. For Your Information 

2. St. ..dardi zed Tests • * 
. 3. • Holland Self-hlref ted Search 

k. SRA MateMals 

5 . jSe 1 f -Asses smenfe 4 ' V 

6. # The Year Two Thousand - # ^ 

7. Personal Goal Setting* f * 

~ 8. A<jtft>n Pl^n t'« 
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* \ , 

The utilization of\the Ne^ds Research and Personal Position 

Audit was determined by an exam! nat Ion" of Group Guidance Report* and 

\ * 
Studenjt Portfolios. The former were reporting forms intended to 

) 

aid In the Counselor-Coordinators' evaluation of curricular content , 
student attendance. and student participation as they related to the 
Guidance Groups. The latter were folders containing the students 1 
completed work in the Career Guidance Unit. The Group Guidance Reports 



were examined in order to determine whether or not 



Coordinators reported each of the 8 instructional a 



the Counselor- 



reas as having been -* 

implerAented. A random sample (roughly t5%) of the Itenth/efeventh graders*. 

t I * 

portfolios -was examined in order to" determine how many of the Instruc- 
tional units were represented by corresponding student worksheets. 
In a majority of cases (67%=) the Group Guidance Reports were 

not completed and returned to the developmental staff of the Career 

o 

Guidance Unit by* the Counselor-Coordinators. None of the Student 
Portfolios was found to be complete, 30% were reasonably complete 

(75% or more of the materials were present), 50% were incomplete (0%* 

♦ / * 

50% of the materials were present) > and 20% were'riot available. In 
total, a majority (£5%) of the materials necessary for completeness 
were missing from the portfol ios . Due to inadequate, and in some - 

cases nonexistent, documentation it was impossible to determine t the 

- • * ■ * * * . / 

utilisation of the Needs Research and Personal Position Audit. 



Task 3 - Survey Staff and Student Attitudes Toward 
Transactional Activities. .The intent of the #ansactional Activities 

r ' ■ ' ' I • , 

•component of the Career Guidance Unit has been described (in Task 

* i 

< » . « 4 

12dAl) as the involvement of tfte' pare icipat'ing student "with the 




widest array of significant peoplel, objects, and events In volunteer 
service In a social community agency." Students who chose to become 

Involved in this aspect of the program enrolled in a Life Skills 

i * 
Specialization. The student who did enrQll in these courses then 

entered Into a contractual agreement with his Counselor-Coordinator 

and a representative of the agency where he\ elected to partlcpate in 

voluntary service. 

• ™ ■ - \ 

"Student evaluations of Life Skills Specializations were obtained 

from the Student Opinion Survey (Appendix C) . The Agency Representatives 
responsible for the instruction and supervision of Academy students 
enrolled in Life Skills Specializations were asked to evaluate their 
student(s) on' a Student Evaluation Form provided by the Counsel or- Coor- 
dinator who was given the responsibility of conducting the Life Skills 
Specialization program. These forms asked for an evaluation of the 
student on criteria such as attendance, promptness, acceptance of / 
responsibility, assistance of others, and the demonstration of 
Initiative. 

When students were asked to rate their Life Skills Specializa- 
tions from 1 (Very Poor) to 5 (Very Good) the mean response of the 
6 studfents then enrolled In Life Skills Specializations was 3.5 (gen- 
erally positive). The small number of students actually enrolled in 
this component of the Career Guidance Unit ln-the Fifth Quarter made 
It impossible to generalize what sort of response would be provided 
by a greater student enrollment of known representativeness. In the 
Sixth Quarter the Student Opinion Survey was ndt administered. Since 
no appreciably greater number of students enrolled in Life Skills 

/ 
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Speclal Izat ions In the sixth quarter, no special attempt was made to 
obtain student evaluation. 

The ? ,data obtained from the Student Evaluation Forms made It 
apparent that the Agency Representatives had positive associations 
with their students, However, the small number of students enrolling 
and being evaluated for their participation, In the program made 
it impossible to evaluate the Instructional staff's perceptions of 
their Life Skills Specializations students meaningfully. 

Task k - Evaluate Student Outcomes of Placement Programs , Infor- 
mation regarding the student outcomes of placement programs may be 
found In the Summative Evaluation section of this report. 

Task 5 - Analyze Case Studies of Student Career Development . In 
order to obtain a subjective measure of student perception of the 1 
expressed goals of the Academy program, a case-study format was decided 
upon as an appropriate evaluation method. It was felt that the use 
\ of a case-study approach might uncover programmatic strengths and \ 
weaknesses which would otherwise go undetected by the more traditional 
objective measures commonly used in program evaluation. It was 
dejcided that the approach used would encompass demographic data, 

s\!anda/dized testing results, curricula data (courses^grades , attendance, 

\ 1 , ' 

«t]k.) and a structured Interview for each of the students studied. 

\ The results of the case study interviews will be presented In 
two forms. First, the data from a 15* random sample group wl II 
be analyzed and applied to the three units of the Academy program, 
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l.c , Basil Skil ls,. Career Development and Career Guidance. In 
It ion, data concerning general student attitudes and opinion 
toward the Academy experience overall will be^summarl zed* Second, two 
case studies will be presented in narrative form encompassing data 
from the Case Study interview and pertinent academic, demographic 
and attitudlnal data which are available from student records. The 
evaluation of the case study interviev/s indicates that there was 
strong student support of two of the Academy program's three uirfts, 
Basic Skills and Career Development; and negligible support for the 
third, Career Guidance. The various ways in which students per- 
ceived the Academy experience to be, valuable, or in some cases irrele- 
vant, will be discussed below by instructional unit. 



Basic Skills Unit . The random sample students who participated 
in tfts case study interviews felt that the most beneficial aspects of 
the Basic Skills Unit's Individual ized'Learning Center involved their 
learning to work at thei r own pace and learning to accept respon- 
slblltty. A majority of the students interviewed Indicated that they 
viewed 'their ILC experiences as help\ul in what they are rioln<j or plan 
to do In the future and a significant percentage of .the student*. IV It th 
the ILC was the best part of the Academy program. In licjhi ol thr 
indicated support of the activities of the Ba^SHUs Unit !>y 
these students and the measurable increases in ReaiJina jh<\ Arithmetic 
scores as evidenced by the Comprehensive Test- of Basic sk^Hs (s<-<- 3 
Special Report 3> Interim Evaluation Report) it can be said that x the ' 
(Individualized Learning Center program was a valuable component of the 
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Academy experience. 

**t Career Development Unit . A majority of the Interviewed students 

'Indicated that they perceived the Career Exploration program as the best 
part of the Academy program. Rather than presenting them wl th vicarious 
learning experiences, the explorations gave the students experiences 
in what they described as the M real world. 11 This comment was pervasive 
in the interviewed students' evaluation of the Career Exploration program. 
In addition, students indicated that because of their exposure to 
different sorts of work environments they were able to make definite 
appraisals of whether or not they would enjoy working In such an envir- 
onment. Majorities of the interviewed students named thejr Career 
Explorations as the most interesting and/or the most worthwhile of their 
Academy activities. A general feeling one received from the case study 
Interviews is that in many cases the career explorations were seen as 
the Academy p^rogVam. In terms of student support the tareer Exploration 
program appeared to be strong and viable. Student comments concerning 
the Career Specialization program reflected confusion primarily. 

Career Guidance Unit . Student appraisal of the Career Guidance 
Unit was marked by apathy, and in some cases, hostility. Of those 
who Indicated that there were some things they did not like about the 
Academy program, 75% indicated that they did not like their Guidance 
Groups. Nearly half of the interviewed students felt that they learned 
nothing as a result of their guidance activities. Two-thirds of the 
"interviewed students felt that the guidance activities had been the 



least worthwhile of all the Academy act i vl t ies, and half of the 
students who suggested improvements in the Academy program 
Indicated that those improvements should be made in the guidance 
activities. Specific complaints registered by the students 
deait v.'ith the perceived irrelevance of the guidance materials to 
the other* components of the Academy program and disorganized guidance 
groups. It Is evident from these sorts of responses that the guidance 
activities must be re-examined, with student input wherever possible, 

I 

In order to make them mqre germane to both programmatic and student | 
needs. | 

\ 

Overall Academy Experience . Student opinion of the overall \ 

\ 

Academy experience was generally favorable. The interviewed students 
felt that they were better equipped to handle work experiences because 
of their participation in the Academy program. A majority of the 
students felt that their career goals and interests had changed as a 
result of new information that the program had provided. Only one 
student indicated that he would not re-enroll In the program if he had 
it to do over again. The students who indicated that they were not 
sure if they would re-enroll generally felt that they would if the 
problems they discussed were attended to. 

Student 1 . Marian is a Black female student who 

enrolled in the Academy for Career Education August of 1973 as a 
tenth grade student. Her reasons for enrolling in the Academy pro- 
grarn included the desire to learn more about different careers, to 
participate in learning activities outside of "school and to receive 
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more individual attention than she was afforded in her previous 
school. Although neither of her parents completed high school, 
both did attend for a number of years before dropping out. Marian \ 
ijas decided that she will continue her education after graduation 
from the Academy and pursue a full-time career in Business Admin- 
istration. 

Results on the Comprehensive Tests of Basic Skills place 
Marian well below the national norms for tenth grade students in both 
Reading and Arithmetic. Pretest scores which were obtained prior to 
her enrollment in the Academy reflected grade equivalents of 7.9 end 
3-2 in Reading and Arithmetic respectively. Posttest scores from the 
instrument administration in April 197A were 7-5 and 6.7 in Reading 
and Arithmetic respectively. Marian's above average grades (B+) and 
attendance are indicative of her perseverance in the Academy program. 
While there was no significant change in her grade equivalent level 
in Reading as measured by the Comprehensive Tests of Baste" Ski 1 1 s ; a 
significant increase in grade equivalent level did occur i|i Arithmetic 
where Marian went from 3.2 to 6.7, an increase of 3-5 grade levels. 
In the case study review Marian repeatedly commented upon the Individual 
ized Learning Center's positive impact on her grades in English and 
Mathematics and stated that the ILC was among the most worthwhile ' 
activities she had been involved in while a student at the Academy. 

Marian has indicated her satisfaction with the Academy program, 
with the exception of Guidance activities. These activities she feels 
are not relevant to the experiences she has had in the other elements 
of the Academy program. In the current school year (1973-1974) Marian 
has participated in career explorations in the Communications and 
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Government clusters one full school day each week and has indicated 
that her experiences In these have been interesting and useful in 
making plans for the future. While enrolled in the. Communications 
cluster in the Fifth Quarter, Marian attended employer courses at the 
KYW television station, the WFLN. radio station and Chilton Publishing 
Company* Her experiences in the Government cluster involved explora- 
tion of the Civil Service Commission, the United States Department of 
Health, Education and Welfare, the InterPal Revenue Service and the 
Marine Corps Supply Depot. In the Government cluster Marian maintained 
an A average in each of the employer sites and had an attendance 
r*te of ]Q0%/ 

Marian has exhibited academic and personal growth while enrolled 
in the Academy program. It is anticipated that the next 'two years in 
the program will build upon that growth and provide her with additional 
insights into her own capabilities and the opportunities available 
to her upon graduation. 

Student 2 . Joe enrolled in the Academy for -Career 

Education in the beginning of the 197.2-1 973 school year as an eleventh \ 
grade student. At the end of the current school year (1973-1974) he 
will "graduate from the Academy program. Joe is a White male student 
whose father completed elementary school and whose mother attended, 
but did not complete, high school. The major reasons Joe has given 
for enrolling In the Academy program were his wanting to get into a 
program which war different from regular schools and his perceived need 
of greater individual attention than he was receiving tn his previous 

EKIC ,;, 
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school. After graduation from the Academy Joe plans to work full-time 
in the field of restaurant management. 

In his five quarters of Instruction In the Academy program (3 
quarters in 1972-1973 and 2 quarters to date in 1973-1 97 1 *) Joe has 
compiled a C average In his courses. His grades range 'from A" s in 
elective courses such as Creativl ty, Hiking, and Workshops in Living" 
(each of these courses were taken in 1972-1973) to F's in English 
^courses and one art course. The majority of his grades have been 
C's. Results of the Comprehensive Tests of Basic Skills Indicate that 
Joe is^>elow th/ national norms for twelfth grade students Jn both Read- 
ing and MathemaVics. Pretext scores which were obtained prior to ^lis 
enrollment, in the Academy reflected grade equivalents of 10.4 and 7.7 
in Reading and Mathematics respect! vely* Posttest scores from the 
instrument administration in March 1 97^ were* 10,8 and 9.2 In Reading 
and Mathematics respectively. Joe commented in the case study interview 
that "learning was easlter" in the Individualized Learning Center than 
It had been in hls^Jrevious school and that he enjoyed wording at his 
own pace. Joe ! s i ncrease i n mathema t ical grade level (1.5 grade levels) 
as measured by the CTBS is seen as substantial. It is Interesting to 
note that while he asserts that he enjoyed the I LC experience and felt 
that he learntd in It, Joe does not feel that the experience wl 1 1 be of 
help in what he plans to do. This attitude was evident throughout the 
Interview, while he enjoyed the program he does not fee! it was instru- 
mental In the formulation of his future' plans . The one area In whirl). 
Joe feels, he has grown as a result of the Acadepiy experience Is In his 
acceptance of responsibility for decisions pertinent to his future plans. 
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When asked if he would enroll In the program if he had it to do over 
over again, Joe replied that he would. 

Joe's experiences in Career Explorations, from his initial 
-woUment to the end of the Sixth QifSrter, included participation 
in the Utilities, Health, Research, and Communications clusters. While 
enrolled in these clusters he explored the Philadelphia Gas Works, the 
Bell Telephone Company and the Philadelphia Water Works (Utilities 
Cluster); Hahnemann Hospital (Health Cluster) ;. Research fo r Better Schools, 
Incorporated, University City Science Center and Wye th Industries 
(Research Cluster); and KYW Television, WF1N Radio and Chilton Publishing 
Company (Communications Cluster). His best grades in- these courses 
were in the Utilities cluster ^B) where he also had his best career 
exploration attendance rate (903;). 

*-CJoe appears to have progressed through the Academy program with 
minimal sel f : invol vement in its expressed goals. In spite of this,'" 
there have been genuine indications of personal growth and a willingness *" 
to accept- responsibility for his future. While his current interest 
In restaurant management is not directly related to an exploration or 
specialization experience, his mature outlook (evidenced by his applica- 

t 

tion to a managerial course) may be seen as an outgrowth of his contacts 
with certain Academy staff members, especial ly -those in the guidance ^ 
unit. 

Basic Skills Unit 

Overview . The following tasks were performed in the evaluation 
of the Basic Skills Itylt in FY 4974. 



1. Survey. Staff and Student Orientation to ILC Procedures; 

2. Determine I ndl vidual izat ion> Participation, and Student 
Progress In Basic Skills; / 

3. Evaluate Adequacy and Integration of Documentation Systems; 
and 

4. Ascertain Student and Staff Opinions of Materials Used. ■ 
Each of these tasks was documented In Task Report ItCI,. Following 
are the summarized results of each of each of the evaluation tasks. 

Task 1 - Survey Staff and Student Orientation to ILC Procedures . 
Orientation for studdnts occurred as a continuous, integral part of the 
Instructional process In the lndlv?dual |zed Learning Center (ILC) 
until the students demonstrated the capacity to work independently, 
within the context of the individualized curriculum. After Introducing 
students to the materials and procedures, staff recapped the procedures 

when necessary and reinforced appropriate use. This orientation was 

/ * 

0 

conducted mainly In small group presentations and Individual conferences 
led by the Coordinator of the ILC. Orientation for staff was also 
a continuous, integral part of the instructional format. Staff were 
Introduced to Instructional techniques which are effective In the context 
of Individualized Instruction: among the methods. and skills especially 
emphasized were planning, tutorial Instruction, and the adaptation of 
materials to the individualized approach of the materials. 

Task 2- Determine Individual izatlonf Participation, and Student 
Progress in Basic Skills . Individual izat Ion of instruction In the 
Basic Skills Unit was determined by three methods; an Individual izatlon 



Checklist which was administered to four members of the Career 

4 

Education Program not directly Involved In the ILC, elicltation of 
student opinion, and the identification and charting of curriculum 
materials being used \>y each student at any given time. .Responses^" 
to the Individualization Checklist indicated agreement that: lessons 
were planned Individually; tests were used to determine levels of 
achievement and to guide use of individual learning units; and students 
were allowed to progress at their own ratfe, were assigned different 
tasks at a given time, were given help Individual ly, and were tested 
individually when they completed learning tasks. 

Student opinion df the ILC activities indicated that a large 
majority of the Academy students perceived the program as allowing \ 
them to progress at their own • rate nd that a majority of students 
fait that the rraterlajs were providing for their individual needs. 

The th i rd 'measure of Individualization was the % identif Icat ion. and 

m 

charting of the level of curriculum materials being used by each student 
at any given time. The use of a great many levels of curricular 
materials by students enrolled in the* ILC would be further evidence 
of the program's Individualization. The results of the identification 
and charting were as follows. Eighteen different units of ILA 
Commuri ftatfons Ski 1*1 s material were being used by 31 students; 23 



different unite of ^LA Mathematics materials were being Vjsed by 77 stu- 
dents. Sixteen different non-ILA English curriculum areas were being 1 > 
used by 64 students; 10 different non-ILA curriculum materials wer© 

A 

being used by 30 students. Overall, 77 different curriculum units 



were being used by students participating In 202 learning activities. 

# 

These figures represent the status of *HC materials in uMC^yring 
the week of April 15, 1 97 1 * • an arbi trari ly selected date. The evaluation 
of the ILC's instructional materials revealed them to be highly indivi- 
dualized. ! ^ 



\ 



Student participation in the Basic Sk, ^ s Un}t ** as determined 
by an examination of student enrollment and rates of attendance in 
the ILC. Student enrollment in the ILC was approximately 90* throughout 
FY 1974. Rates of attendance in the ILC ranged from 70$ to. 80* by 
•quarter.! A complete discussion of rat^£ of attendance may be found ^ 
in Task Report l^Cl . 

Student progress in the Basic Skills Unit was^eval uated by 
examining the number of skills mastered in the ILA curriculum, the 
quantity of credits earned, grades earned, and progress on* the Com- 
prehensive Tests of Basic Skills (CTBS). The last (progress on t*e 
CTBS) will be discussed In the Summattve Evaluation section of this 
report. In the Fifth and SUth Quarters the average numbers of skKlls / 
mastered in ILA Math was approximately. W for tenth/eleventh grade 
students, and Ik for. twelfth grade students/ In ILA English during 
the same period t&e average umber of skills mastered\was 15 for tenth/ 
4 eleventh grade students and 12 for twelfth grade studehtV. The Director 
of the Basic Skills Unit reported that instructors tended to use non- 
ILA materials more for English than for mathematics activities; this, 
might explain the differences in skill mastery in these areas. Skills 
mastered information must, be Interpreted with* caution since discrepancies 
could be due to factors such as progression to a new curriculum, the use 
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the use ofnon-ILA materials and student tootivatton. 

The average credit earned per unit In the 1LC Math, actlvl t le*/ * 

was 'approximately .24 In the Fffth tfnd Sixth Quarters for tenth/. 

eleventh grade Students and .28 foV twelfth grade students. In ILC 

English activities during the sameperlpd tenth/?l€ventl% grade students. 

earned an average of .29 credits while the twelfth grade students yarned 

an average of, .27 crests. The average grade earned In ILC activities , 

by Academy, stiidents was in the* C to C+ range. 



> 



Task 3 - Evaluate Adequacy and Integration of Documentation 

Systems. The documentation system .used in ^he ILC consisted of * manual 

system which was fully developed 'and a computerized system fihlch is 

still in the developmental stages. The manual documentation sysVfca^ 

consisted of fou reforms: the General Information Form, the^ Student 

Prescription Booklet-Form, the Student Profile and' the Student Summary 

Shee>w Each of these forms is brlefly x descrlbedbe?ow* 

^he^ GeTrecal Information Form was filled out for each student upon 

his or her entry Int^the ILC. The form was composed of three parts. 

Part one Included general per^o«^l information concerning the student 

.such as sending school, date of bl rth ,>}^de^1eve1 , 'parent or guardian, 

home tefephone-^l^erj^nd emergency telephone ftttmber. The second part 

of the form contained the results of the student's performance' on 

standardized tests. The third part of the form served as a learntng^ 

* 

contact between the student and his instructor. Basic skills objectives ^ 
for each subject area were agreed upon in a student-instructor conference; 
this agreement was recorded and signed by both the instructor and thev 
student. The^eneral Information Form was placed In the student's file 

• in 4 



and cOuld be referred *to*or updated dur I ng «the 'school ycar^ 

The Student Prescription* BooklefrForrn provided a record of the* 

student's progress thrQugh Individual skiM area*. These -forms were 

... °* , *, r > 

compl^tnd dally, and are housed l« each student^ work folder; 

^ , * TK» Student Profile recorded the student's progress In units.qf 
tft$\ \\A communication skills or mathematics curriculum materials* The , 
"form. was not used fpr. students who were not participating in either 

subject area of the curriculum. The Student Profile was stored in 

♦ * * % ,? * 

each student's* work folder. u/ 00 * * 

- - ' *• / ' - 

The Student Summary Sheet was filled out quarterly for v each 

*t-#f* , ■ ^ ' ^ * * 

student* "The form contained Information regarding the student's 
Attendance* grades , total hours scheduled; total hours attended, late 
arrivals to learning activities, and the particular work scheduled * 
and mastered by the student. This Information was recorded by the * 
teacher for each subject area In the Basic Skills Unit , * 

The staff and administrator bf the Academy for Career Education * 
reported that these form} had proved adequate for the needs of the V 
instructors and student*. The manual system also proved to be adequate 
for the needs of the evaluation staff In completing tasks regarding the 

e * ■ 

Basic Skills Unit* Howeve^, there was a time element which proved to 

be cumbersome for both the .Academy staff and thie evaluation staff* Corn- 

pfetlon of the Stu^fht Summary Sheets took as long as two weeks after . 

• / 
the end -of *a quarter. % This delay plus the processing time necessary 

has made It difficult to report out results with the speed that would 

7 ' 

make them most useful . ' . 
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The computer system o^ documentation was Intended to provide 
all forms of Information contained b/ the manual documentation system 
and enable additional analyses. The initial plan called for. the 
implementation of the manual system at the start of the academic" year; 
this was implemented on schedule, with Weekly Attendance Reports \ 
and Student Summary Sheets designed by the evaluation staff being used). 
The automated system was to be designed during the Fifth Quarter, field 
tested Sixth Quarter, and evaluated relative, to the manual system in the 
Seventh Quarter, fn fact, given the complexity of the computer system 

\ • - \ 

required and the expense involved, ( i t was not possible to implement the 
automated system until the Seventh Quarter. Even at this late date, 
the system was pot complete; further development is required. The 
task of' comparing the twp systems (manual vs. computerized) was thus 
not accomplished. It is apparent that the automated system still has 
disadvantages which cause interference in the operation' of the learning 
center. It also seelns clear that the automated system hjs much potential 
fpr instructional and evaluative: purposes. The real izat ion of this 
potential/ wi 1 1 requi re the japplication of more resources. in systems 
development. Since it is unlikely that >uch resrturcos wi 1 I be uv»j»lable 
in the program budget, special Quoding iiuy be sought. The area of 
computer appl icat ions faci 1 i t<il i nq iwli vidual ; /dtion is seen as an 
important one; however, it must be fit into rhe 'schema of priorities. 



Tas k k - Ascertain Student and Staff Opin i on of Materials U sed . 
Student responses to the Student Opinion Survey (Appendix C) am! «m 
informal evaluation survey indicated that about one third of the 

r f ' 
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Academy students found the I LC materials interesting/ while a plurality 

/ " 1 

considered them average. A majority of the student^ found the mafter- 
ials to be above average in terms of their adequacy, while a small 
percentage {\5%) found them to be below average in this regard. A 
^majority of the students (approximately two-th i rds) t nd tcated that 

they felt they were learning more in their ILC classes than they had 

~> 

in their other school/s. / 

J 



Staff opin ion, obtained through personal interviews, was 
highly positive toward the materials 1 ability to provide individual- 
ized instruction fq/r the students. However/, they indicated a desire 

/ 

to # have a new diagnostic test for determining more specifically the 
weaknesses of individual students. The variety and extent of the 
different materials were deemed adequate land plans to utilize newly 
developed materials were discussed. The/ staff did indicate that the 
interest le^el off the materials could b£ raised. 

i 

Utilization of Evaluation 

The evaluation activities outlined in the Introduction Section 
above may serve as an initial functional definition of the role of 
evaluation within the project. These activities were: formative eval 
uation, data sys tems\ deve lopmer^, sunmative evaluation, instrument 
development and cooperative research. OfL these major evaluation 
lactivities, the first \two were designed primarily to be of benefit 

|to project staff, while v the last three were oriented more toward o 

I 

external parties. The tale of evaluation and utilization of evalua- 
tion results will be discussed within this context. 
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ln formative evaluation it was attempted to assess each pro- 
gram component as defined in the "Operating Plans for FY 197V 1 
after the component design and field test cycles had been completed. 
The intent was to provide formative data for use in revising the 
components for production of replication specifications and program 
performance testing. The role of evaluation was thus to provide 
independent assessment as an input for program development. The 
results of this activity were also intended to have use in program 
monitoring and management decision-making. 

Tn data systems development the evaluation staff designed 
and monitored a system aimed at consistently and accurately capturing 
the most significant operational data on student performance and 
learning activities conduct. This system was first implemented a-s 
a manual process whose subsections were gradually replaced by 
machine processes. It was intended to produce a tested, fully 
automated system which could subsequently be operated without exten- 
sive technical assistance from the evaluation staff. The information 
gathered and treated by these systems was intended to have relevance 
to project operations by obviating clerical drudgery and to project 
management by providing extensive data for monitoring 1 and declsion- 
mak i ng . 

Both of these aspects of evaluation proved to be valuable. 
Both proved valuable in actual utilization in other than the ways 
Intended. A discussion of the discrepancies may prove beneficial 
in gaining a realistic view of the actual role of evaluation In 
projects such as this one- 

71 



-68- 



Th e formative evaluation objectives related to program development 
and Management utilization were met only to a limited extent. This 
seemed to be true for a variety of reasons which are complex aritf 
interrelated. In a prototypic program with multitudinous curriculum 
elements, many of which are developed as the program Is operated and 
also which shape themselves to fit individual students' needs, specificity 
in evaluation is difficult to attain. The choice of assessing myriad 
individual curriculum pieces or abstracting and generalizing is often 
faced. In the corrtext of limited resources and broad demands the 
latter choice becomes attractive, if not necessary. Generalization Is 
also increased when program elements are lacking in prespeci f iable detail. 
This results in evaluation findings which are generally Interesting but 
not specifically applicable. 

Related to this is the generally prevalent time-pressured flow 

of both development and evaluation activities. One cycle or year over- 

J 

laps the next without a break for reflection, which might allow the 
considered application of evaluation findings and suggestions to develop- 
ment and visa-versa. In effect, there is not a design, field test, 
evaluate, redesign or other Sequenced development configuration. All 
processes are conducted simultaneously in a general movement toward 
improved effects. This situation may be conducive to the continuous 
operation of a demons t rational program. It ^an be responsive to 
the participants in the project. But it cannot foster maximum utiliza- 
tion of development or evaluation efforts. 

Together, the practically necessitated level of generalization in 
evaluation design and the limited opportunity to sequence development 
and evaluation In real life mitigate against the ideal utilization of 
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formative evaluation. Formative results seem to have more usefulness 

in project monitoring and management, but here also the level of abstract 

tion and time sequence present problems. 

The abstraction problem can be sol vecJ only by prespec i fy i ng at 
a management level what information is needed and then designing a 
system to get it. This devolves on the second major aspect related to 
internal utilization, data systems. 

Good data systems in comprehensive educational projects relieve 
operational staff of clerical duties as an incentive for them to comply, 
provide operational staff with useful information they otherwise wouldn't 
have, generate consistent and accurate information, enable rapid feed- 
back to interested parties, and establish a data base for project 
evaluation as well as broader research. They are difficult to construct, 
more difficult to implement, and even more difficult to complete as 
independently functioning products. Such a level of completeness is 
seldom reached within the context of an experimental proj ect. System 
components are usually effected at varying levels of sophistication. 
In this project all systems were designed and completed with manual 
input; machine input was designed in all intended areas and field 
tested in some. At this stage the functioning of the system is limited 
in several regards. Of primary relevance to this discussion, rapid 
feedback for project management is hampered. This often resulted In 
findings which confirmed already formed opinions. Such confirmation and' 
specification has value, but it does not fulfill the objective of early 
warning about problems. 




Summarizing the discussion above, It must be concluded * that evalua- 
tion has had only limited success in informing program development and 
project administration. There was evaluation utilization in both, but 
it cannot' be characterized as a key ingredient. 

The question then becomes what is the effective role of evaluation 
in interaction with the project staff? Not to be urfduly harsh, or * 4 
wors$, inaccurate, it can be said with justification that the role of 
internal evaluation has real significance for project^ltaff and^ conduct. 
That role, however, appears to be different in real life from what it 
might be fn idealized characterizations. 

In order to, add objectivity to this discourse on the real 
utilization of evaluation, a survey of key project personnel /was con- 
ducted. Confirming the points above the'staff indicated' that the 
evaluation findings were very "interesting" (*».3*» average rating out 
of a possible 5.00). They were found to be substantially less "helpful 11 

(3.20 average rating}, and less an agent of "change" (3.00 average 

* 

rating), The primary impact of internal eval uation was seen b^ the 
staff a*s residing in the following: Evaluation reports conslsely and 

accurately depicted the status of the project components; in this 

' * 1 

documentation function they provided realistic information on what was 
going on and how well program elements were functioning. Evaluation 
results served to confirm problems pnd successes which had been diagnosed 
at fn Impress ional , non-quantitative level and identify new ones. In 
this diagnosis function they helped to define the student of known 
phenomena and to uncover new phenomena. The very existence" of ah 



evaluation effort served to keep people "hones* 11 by establishing 
the means for Independent review and assessment. Thfs discouraged 
overstatement of progress and understatement of problems. Evalua- 
tion reports were also viewed as Importantly virtue of thefr v 

ful fil 1 IngNcontract requirements with the funding agency, NIE. Finally, 

> m 

although it Is difficult to ascertain degree of causality, the evalua- 

tion findings were seen as playing some role In the redesign of 

program elements, particularly the guidance and „ supplementary components 

Thus, staff saw the evaluation effort within the project as being 
useful in documentation, diagnosing and illuminating problems, ennanc 
ing accountability, ful f i 1 1 ing ^contract requirements and suggesting 
areas for program change. The value of such utilization compared wi th 
the ideal may be debated. In the context of this project it appeals 
to be both real and significant. Since existing research literature 
seldom'discusses the actual utilization of evaluation or its interaction 
with project staff, the general i zabi 1 i ty of the concl us ions . presented 
above cannot be known. These conclusions are herein elucidated in hopes 
that more consideration will be given to evaluation as a contributing 
resource in project development in addition to a technological endeavor. 
Factors seen as fostering growth toward the ideal in evaluation utiliza- 
tion Include sequencing the developmental cycle to allow incorporation 
of evaluation staffs to the mutual contributions possible, defining the 
'evaluation scope to permit specificity on priority curriculum elements, 
and providing resources to enable rapid feedback data systems. 

The three major evaluation activities more oriented toward external 
parties - summative evaluation, instrument development, cooperative 
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research - exhibited a different profile during FY. 1971*. These 
activities were generally more complex technically than those geared 
<to the Internal project staff, but their utilization was simpler 
and more conforming to original objectives. 

In summative evaluation it was attempted to assess the effects 
of the program overall on its participants. The intent of this effort 
was to estabUsh the educational significance of the program, to pro- 
vide Information useful in program dissemination and expansion, and to 
determine areas in need of longer term development. 

The function of instrument development was to provide the means 
for appropriately testing effects related to summative evaluation. This 
effort was intended to have use- for the present project as well as 
other studies in career education. ' « 

The cooperative research activities were initiated this year 
under the guidance of the NIE evaluation offices for EBCE. The 
evaluation staffs of each of the projects participated in joint confer- 
ences which, in addition to other evaluation concerns attempted to 
isolate research issues related to EBCE and to pursue them as an activity 1 
beyond contract commitments. Results of this effort were intended for 
dissemination to the research community. 

In each of these major activities considerable success was met. 
The summative process and results are characterized by the present report. 
The instrument development is reported in the appendices. The cooper- 
ative research effort is more .nascent , but it has resulted in professional 
•papers and development though* which help to keep the whole evaluation 
effort more vibrant. 
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Each of these activities has resulted In products of use primarily 
to parties external to the project. For NIE they certify, the conduct 9 
of the project, enhance the credibility of results, and serve as 
independent products of the evaluation staff. They provide public 
relations Infortnat ion for potential adopters and other interested 
agencies and Individuals. They also have use in the ext$ntion and 
development of existing programs. This group of activities, then, is 
not subject to the intricacies of implementation .and utilization 
discussed above in regard to the more formative endeavors. The project 
staff saw these functions as perhaps the most important for evaluation 
to perform. 

In summary, the evaluation effort produced results in fiv£ areas 
of functioning which served to define the perceived role of Internal 
evaluation. The objective least well met was in the utilization of 
findings in program development. It was suggested that both evaluation 
and development personnel need to work on facilitating procedures to 
overcome the impediments inherent in this type of demons t rat lo/i project. 
Data systems development fell short of the Ideal envisioned, but met 
the con tract ^and operatiorlal needs. Project monitoring seemed to func- 

tlon adequately. Summative, instrument devjelopment and cooperative 

* * 

research activities proceeded very welh From this It may be concluded 
that a strong evaluation effort was designed and implemented. Results 
were utilized in numerous way$, most of ^them meeting their objectives 
in utilization. It is suggested thaj^further attention be given to 
the actual role of internal evaluation. Desired* roles which are not 
practible should be either discarded' or made realistically possible by 
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revising project cycles or processes. They should not be carried 
slroply out of obedience to tradition. Other roles which are tradition- 
ally peripheral but seem to produce results in real situations may be 
expanded. Actual functions and Interactions should be "documented and 
Investigated. In this way development in the application of, evaluation 
technology can be furthered. 

\ 

IV. SUMMATIVE EVALUATION 

i 

* J* 

The Intent of the summative evaluation efforts of the FY ]37k RB^ 
Career Education Program was to determine program effects on participants 
the cost feasibility of the program, and the marketability of the 
developed program. This section of* the Final Evaluation Report will dls- 
cuss each of these as well as design and population Issutes which 
relate to conclusions presented. 
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Evaluation Design 

Student Groups . Jhe general design of the summative evaluation 
efforts reflected the developmental status of the RBS Career Education 
Program. FY 1974 was the second year of implementation of the Career 
Education Program; as a result two distinct samples of experimental 
students were simultaneously provided educat tonal- services. These 
groups were the 

ACE Group - Students or igi nal ly recruited for FY 73 proqram, 
all equivalent^ 12th qraders, all In program for o 
second year. 

ACE-Olney Group » Students recruited for FY' 7k program in cooper > 
ation with Olney^lgh School, grade equivalent split between 
10th and fl th graders, all In program for first year. 

TO 



The ACE group has* no appropriate control group* Two groups- 
were selected to serve as non-equivalent control groups for the ACE- 

• * 

Olney 'group. These groups were the 

Comparison Group - Olney students who appHcd for ACE-Olney 
program, were accepted, but finally decided to 
• not p.nroll in the program. 

Context Group » A random selection of equal numbers of K)th, 
llthjj. and 12th graders from Olney, no known exposure • 
to program, no intended selection biases. 

The Comparison group was viewed as a control group for interest in, 

but not partlci potion 'in, a Career Education Program." However, this 

group self-selected itself out of the ACE-Olrtey group prior to the 

beginning of the FY 7** program. This sel f-selfectlon cannot in any. way a 

be considered a random selection; this n on random self-selection creates 

a possible unknown selection bias irt the comparisons 1 ' involving this 

group. The Context group was viewed as a legitimate comparison group 

for determining theeffects of a trad! tional .school program for 

comparison with the Academy program , - Since these students were not 

selected for equivalence to the Academy students, such comparisons 

must be regarded as. gross indications. However, the random selection 

should ensure this group's representativeness of a traditional high 

school student body, 

\ 

At the beginning of the FY 7** program,- the ACE-Olney, Comparison, 1 
and Context groups were found to be comparable in terms of age, previous 
school attendance, previous grade point average, parertal occupations 
and parental education levels. Data were unavailable at the*beginning 
of the year on the race and sex ch£r£ct^rist ics of the control groups. 
There was a noted discrepancy between tffe ACE-Olney group and the 
control groups in terms of post-secondary plans; the control group 



students had-a ma r kedly hig h e r p r o portion planning immediate entry 
« i ♦ 

into a job than did the ACE-Olney group. Thfs discrepancy in post- 

secondary plans w*s viewed as. possibly affecting later analyses; \ 

however, If it were to be the only dUcrepahoy, i t was felt that 

planned analyses cou^S be performed on the po^ttest data with little 

^^chajjenge to the Integrity of the procedures or interpretations. 

f 4 Sex and race characteristics of -the control groups were avail- 
ed 

able at the end of the project year and were presented in the Student 
Population section of this report. Addi tiona4. discrepancies were noted, 

especially for the Comparison group. The composition of the Comparison 
group was 2 to 1 female while the composition of the ACE-Olney group 
was *3 to 2 female. The racial composition of the Comparison group 
wcs 3- to 2 Black while the composition of the ACE-Olney group was 
. h to 1 Black. The Comparison group contained only 11th graders while - 

\ the ACE-Olney group contained both 10th an^d 11th grade students. 

\ # . 

\ On the average, the Comparison students were one year old,er than their 

4 

\ ACE-Olney counterparts. The earlier noted future planning discrepancy 
became greater with $0- percent d^f the Comparison students indicating 
^n employment orientation while, 18 percent <}$slMt ACE-Olnoy students 
indicated such an orientation. »' 

\ The Comparisoh group appears to be different fnim Liu- ACE-Olney 
groufi on a sufficj^nt number o-f «cont i nuaa to render it bighly question- 
able os a Control group. Xhisl finding will have an effect of tUv 

i * 

explication of the exper imental dqsiyn to b<- •*lscus«!'«*«l Liter ii» this t 

Section. 1 ' *' j v • , 

- ' * / 



Instruments . The following lnst r uments / were administered* to 
both experimental and control, groups. WhUc£ they /have some format iv^ 

* - . ' r ' \* <; 

utility, the! r. primary use was Intended to be sunmaHve,^ < ' 

1 . Comprehensive Tests of Basic Skills (CTBS) - TM* instruniertt 
measures traditional academic skills. The Readlrfq and 

- x >. • 

Arithmetic subtest? were used, yielding the^fol lowFng scores: 

Reading Vocabulary, Reading Comprehension, Reading Total 

Arithmetic Computation, Ar J thmetic Concepts , Arithmetic 

Applications, and Arithmetic Total. Each of the scores 

is fc^ the form'of standard iscale) scores. The instrument 

has been well developed and documented, but is subject to 

the usual insensitWities of standardized instrunrentsr 
v 

2. Career Mat qrl ty Invento r y (CM I ^ - This instrument wajs 
^designed to measure Career Attitudes and a set of career 
competencies: Self Appraisal, Occupational Information, 
Goal Selection, Planning^ and "Problem Solving. . Onl y the 
Occupat ipnal Information and Planning competency subtests 
were used. v 

This instrument Has befeji well developed but ft ha$ not 
been extensively researched and documented. The four EBCE 
projects have questioned the reliability and validity of the 
scales which compose the CMI. All projects forwarded their 
Item and factor analysis of the CMI. The results of this 
analysis were no,t available to RBS at the time of preparation 
of the Final Evaluation Report; thus, questions regarding the 
CMI still exist. 
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3. • Asseism^nt of Student Attitudes Questionnaire (ASA) - This 

j;/ "7 — 7 v*: 1 1 1 1 ;o ^ ■ ,/ : * — 7 — - — * — ; 

, Instrument' has* teen developed by the R$&^staff to measure ' * 
, attitudes tqwtfrt s^eraT AieiiieotjVJn the learning, environment :> 
P*\ ^ * \^ \ Educatloh GcferaT^Tgfio&V- Currtcubim, School Resources, and 

. • . • ' '• , * ; * ^> - • - . ,- • 

^ VWJ^V*;; . School Cdunsixlbtq,, *VhiW further de^lppf*ertt, effort Is indlca- 

\« ,\ ted for'tlie^Wtr^jT^ a hljjbly reliable and 

* valid attitude scale," A report of thf reliability and. 



4' 



V , / report 



vaUdfty of the instrument is contained: in the appendix to this 
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Student" Demographic Da fee Queajionnalre (SbQ) \ ThlOrts truijoent 
was constructed by f he* evaluation staffs of all EBCE projects 
to provide common data on basic characteristics. The question- 

I o 

naire Includes: Name, Sex, Birth Date, Race, Grade Level, 

Po^t Secondary Plans, Parents Education Level /Parent OccUpa- 

l 

tions, Sending School Grades, and Sending School Attendance. 
Another series of instruments were established for assessing' 
participant groups only. These instruments have all been developed 
by the evaluation staff. They include the Student Opinion Survey, 
the Parent. Opinion Survey, and the Employer Questionnaire. Descriptions 
and discussions of these instruments are contained in the appendix of 
this report. 

Hypotheses . Hypotheses regarding program outcomes fall into 
two categories: those relating to student outcomes and those relating 
to other program effects. Hypotheses regarding student outcome's' of 
the Career Education Program reflect the existence of the two rflstVhct 
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student experimental samples as well as the existence of control 
groups* Some or the hypotheses are stated 1n terms of significant 
gain with the presenting level the basis of comparison; others are 
stated in termi of relative or comparative outcomes • The former 
relate onfly to the experimental students and the latter relate to 
"this year's entering students and their control groups. ,* 

Hypotheses of program effects are listed below* The instruments 
and respondent groups whi£h are appropriate for'the testing of a 

-v 

V 

hypothesis are 'isted with each hypothesis. 

The hypotheses which relate to student outcomes are. , c 

1. Students will gain significantly (p<.10) in basic skills over 
the N course ol; the year. Respondent groups are the ACE 
and ACE-Olney groups. The appropriate test is the CTBS. 

2. Students will gain significantly (p<.10) more in basic skills 
than comparable students In a traditional school. The 
respondent groups are the ACE-Olney f Comparison and 
Context Groups. The appropriate Instrument is the CTBS. 

3. Students will gain significantly (p<. 10) in career maturity. 
The respondent groups are the ACE and ACE-Olney groups. The 
appropriate instrument Is the CMI. 

k. Students will gain significantly (p<.10) more in career 
v maturity than comparable students in, a. traditional school. 

- m J ' 

* t I The respondent groups are the ACE-Olney t Comparison and 

Context groups. The appropriate Instrument Is the CMI . 

^ 5. Students will evidence a significantly (p<v10) more positive 

» » 

attitude toward school than students in a traditional school. 
The respondent groups are the ACE-Olney, Comparison and 

ERIC _ 
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1 \ 

and Context groups. Th|e appropriate instrument is the 
ASA. 

6. Students will gain significantly (p< . 10) in career knowledge 
over the course of cluster experiences. The respondent 
o groups are the ACE and 'the ACE-C\lney groups; the appropriate 

\ 

instrument is the cluster test of knowledge. Due to 
difficulty in filling the staff position of instrument 

developer, only one cluster test o)\ knowledge was sufficiently 

\ 

developed to be used q'n a pre-post test basis. Another 
appropriate measurement is the Occupational Information 
and Planning competency subtests of tr^e CMI ; they will be 
used to test this hypothesis. \ 
Hypotheses related to other program effects are listed below: 

1. Employers will be able to provide learning experiences 
sufficient to meet student nee,ds and interests. This hypothesis 
will be tested by comparing student need$ and stated student 
interests with the learning experiences provided by employers. 

2. Empl oyers wi 1 1 evidence a posi tive atti tujde and commitment 
regarding the program. The respondent grbup is the pool of 

employers, unions, and agencies providingilearning experiences. 

j 

The appropriate instrument is the Employer! Questionnaire. 

3. Parents will evidence a positive attitude land commitment 
regarding the program. The respondent group is the parents 
of the experimental students. The appropriate instrument 
is the Parent Opinion Survey. 

i 

k. Institutional structures will be established to enable 

i 

che conduct of the program. The implementation of the pro- 
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gram will be examined for inst! tutional structures and 
interrelationships which enable the conduct of the program. 

5. It will be demonstrated that the program can be operated on 

a cost feasible basis. This hypothesis will be tested by com- 
paring actual costs with criteria for feasible costs 
previously defined in the M Operating Plans for FY 197^. " 

6. It will be demonstrated that there is a ready market for 

the program. This hypothes i s wi 1 1 be tested with the - » 

results of a poll of potential adopters of the developed 
career education program. 

Administraf ion of Instruments . The CTBS, the CMI , and the ASA 

were admi nistered' on a pretest-posttest basis to all studertt groups. 

The testing schedule and rationale are presented in the appendix to 

this report. Intertest interval for the CTBS was 8 months for all 

groups. The intertest interval for the CMI- was 8 months for the ACE- 

Olney group and its control groups and 6 months for the ACE group; 

since there are no direct comparisons to be made between the ACE group 

and other groups, this discrepancy is viewed as not substantially 

affecting the posttest analyses. The intertest interval for the ASA 

was 7 months for the Comparison and Context groups, 6 months for the 

ACE-Olney group, and 5 months for the ACE group; any bias due to 

differences in intertest interval is against the exper imental students. 
j 

0 

Overview of Experimental Design . The testing of hypotheses which • 
relate to experimental students only are examples of a one group pre- 



test-posttest design as elucidated by Campbell and StanJey (1963). 
The testing of hypotheses related to the ACE-Olney Group and its 
control groups are examples, of the non-equivalent control groups 
pretest-posttest design. Hypotheses related to parents and employers 
are examples of one group retrospective posttest design. 

Statistical P rocedures . The statistical procedures used for test- 
ing hypotheses related to experimental students only are correlated 
"t" tests. For the CTBS scale scores are used; for the CMI and ASA 
raw scores are used. The statistical procedures used for testing 
hypotheses related to comparisons of the ACE-Olney, Comparison, 
and Context groups are analyses of covar lance wi th the pretest level 
of performance the covariate and the posttest level of performance 
the criterion measure; this statistical icontrol equates the groups 
for pretest level of performance. Due to the doubt cast on the compar- 
ability of the Comparison group, all analyses regarding the relative 

0 . « 

performance of the ACE-Olney grpup are conducted in two ways: (1) a 
three group analysis which Includes the Comparison group and (2) a 
two group analysis which 4ncluclds only the ACE-Olney and Context groups a . 
Descriptive statistics are used as the basis for testing hypotheses 
related to parents, employers, ccsts, and marketability. 

Tes-ttng of Studenfc Hypotheses 

Hypothesis 1. Students will gain significantly (p<.10) In basic 

i 

ski 1.1 $ over the course of the year * The hypothesis that experimental 
students would gain significantly in basic ski 1 Is. prof iclency over the 
course of the year was tested by an examination of gains on subtests s 
of the CTBS. The subtests used were Reading Vocabulary, Reading Compre* 



hensioir, Reading Total, Arithmetic Computat Ion , Arithmetic Concepts, 



Arithmetic Applications, and Arithmetic Total. Correlattd "t" t«sts 
were calculated for both groups on each of the measures; one tail M t" 
test critical values were used since directionality of results was 
contained in the statement of the hypothesis. In all cases, scaje 
scores were used in the analyses since they are the most reliable^ 
and precise scores yielded by the CT0S. For the convenience of the 
reader, pretest, posttest and gain scores are presented in grade 
equivalent form for each of the groups. This Information is presented 



in Table 29. 
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lable 29 CTHS Crade Equivalents 

Experimental Croupi ^ ^ 

c 

ACE-Olnajr Croup ACE Croup 

n •» 38 n • 55 



9 Pre Poat Cain ftm tomt Cain 
heading _ _ _ 

Vocabulary 9.0 O ~" 10.0 .3 

Cowprahrnaion . J.l "1.0 -.1 ^8.9 9.8 .9 

Total 8.2 6.6 .4 *9.3 9.9 .6 



Arithmetic 

Computation ?A 8.1 .7 8.4 t.9 .5 

Concept* 7.7 l.t .9 8.3 9.0 .7 

Application* 7.2 7.9 .7 7.5 8.6 1.1 

Total 7.3 8.1 .8 8.2 ' t.9 .7 



ACE Group . 4 The ACE group demonstrated significant gain on two 
measuf-es'of basic skill proficiency: the Reading Comprehension- 
si/btest and' the Arithmetic Applications subtest. On the remaining 
measure? of 'basic skills development, the ACE group showed some 
gains; however, these gains were not at a level of statistical ygni 
. ficance. CTBS gains for the ACE group are presented in Table 30. 



T*Me 30 ClftS Scale Score Gain* 



Reading 

Vocabulary 

Contprahenalon 

Total 



ACE Cro tp 
o • 55 



Pre Po»t 

59*5.45 596.48 

572.3* 597.93 

532 04 595.66 



Cain -t" 

1.03 0.06 

25.59 1. 34* 

11.62 0.91' 



Arlthautlc 

Computation 536#tO 53ft. 13 1. 33 0. 08 

Concepts 533. 46 548. tl 10.34 0.57 

Application* 515.49 552.07 36.5A 7. 12* 

Total 530.22 540.55 10. 33 0.59 



critical value, t * 1.31, p - .10. df 



fc- AC'E-Olney Group , The ACE-Olney group demons trated significant 
gain on 5 of 7 Measures of basic skills proficiency: Readinq 
Vocabulary, Arithmetic Computation, Arithmetic Concepts, Arithmetic 
Applications, and Arithmetic Total. The ACE-Olney Group also demon- 
strated gain in basic skills assessed by the Reading Comprehension 
and Reading Total measures ; however , the gain was rtot at a level of 

r 

A 

statistical significance. The CTBS gains for the ACE-OTney group are 
presented in Table 31. 



Table 31 CTIS Scale Score Cain 

ACE-Olney Croup 
■ - J* 



fra \ Foit Calo "l" 

Reading * 

Vocabular) 593.5ft S63 97 24. 39 1 .39* 

Coaprahenalon 531.90 543.97 11.07 .71 

Total 532.34 S51.21 18.67 1.13 



Arlthaatle 

Computation 495.97 523.14 27. 9' 1.75* 

Coocapta 50H.45 540.26 31.11 2.15* 

Application* 49S.51 S27.22 U.7J 1.A9* 

Total 492.44 52 3.24 30.00 2.01* 

*t £ i.3i, p -.10. df > 30 
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The results of the correlated "t" test analyses of CTBS measures 
of basic skills gains offer moderate support for the hypothesis that 
students show significant gains in basic skills prof iciency.over the 
course of the year. The support is much stronger for the ACE-Olney 
group which demonstrated significant gain op 5 of f 7 measures used than 
it is for the ACE group. The differential achievement by grade level 
should be a topic for /urther investigation. 

Hypothesis 2. Students will gain significantly (p<.10) more*in 
basic skills than comparable students in a traditional school . The 
hypothesis that experimental students would demonstrate "signf icantly 
more gain in basic ski'lls proficiency than comparabl e students in a 
traditional school was tested by comparing the performances of the 
ACE-Olney, Comparison, and Context groups. The comparisons were 
conducted by performing analyses of covariance on the posttest performance 
of the cjroups on subtests of the CTBS; the entryylevel of the groups , 
was made equivalent through the use of the covari^fice procedure. The 
subtests used were Reading Vocabulary', Reading Comprehension, Reading 
Total, Arithmetic Computation, Ari thmet ic. Concepts, Arithmetic Applica- 
tions, and Arithmetic Total. 

Due to previously defined pon-equi valence of the Comparison group, 
two sets of analyses were performed on the CTBS data: orte included the 



Comparison group and. the other included onl^ the ACE-Olncy and Context 
groups. Each set included an analysis of covariance for each subtest 

of this CTBS. . V 

• X 

In all cases, the criterion measures were the posttest scale (st*n- 
dard) scores ar>d the covariate was the pretes* scale score. Scale 
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scores were selected for the analysis since they are the most stable 
and precise scores yielded by the CTBS . For the convenience of the * 
reader, pretest, pqsttest, and gain score* are presented, in grade 
equivalent form for each of the groups. This information- is 
presented in Tables 32 and 33. 



Table 32 CTBS Grade Equlvaltnc* 
-. Experimental Croup* 

ACE-Olney Croup ACE Croup 

n - 38 a - 55 



Reading 

Vocabulary 

Comprehension 

Total 



Arltbjuthlc 

Computation 

i Concept* 

Application* m 
Total 



Pre 


Poat 


Gala 


Pre 


Poat 


Cain 


1.4 


9.0 


.6 


•.7 


10.0 


.3 


1.1 


8.0 


-.1 


1.9 


1.8 


9 


8.2 


8.6 


.4 


•.3 


9.9 


.6 


7.4 


8.1 


.7 


1.4 


'§.• 


.5 


7.7 


1.6 


.9 


1.3 


,9.0 


.7 


7.2 


7.9 


.7 


7.5 




1.1 


7.3 


1.1 


.6 


1.2 


1.9 


.7 



^ Tabic 33 

CT8S Grade Equivalent* 

Control Groups 
I Conpar laon 



Context * 





* 


n * 46 






i - 5/ 






Pre 


Post 


Cain 


Pre 


Post 


Gain 


Reading 














Vocabulary 


7.9 


- i.8 


.9 


a. 6 


9.0 


.4 


Coaprchantlnn 


7.0 


7.7 


c .7 


7.6 


7.7 


.1 


Total 


7.4 


8.2 


.8 


8.3 


8.4 


.1 


Arithmetic 














Computations 


7.4 


7.6 


.2 


7.5 


7.7 


.2 


Corcept* 


7.7 


7.6 


-.1 


7.9 


1.5 


.6 


Application* 


7.3 


7.0 


-.3 


7.6 


7.0 


-.6 


Total 


7.3 


7.5 


.2 


7.5 


7.7 


.2 




r 

Reading Vocabulary * Each analysis failed to identify a „s i qnl f leant 
difference between the ACE-Olney group and either the Comparison or 
Context groups; in both cases the F value was less, than 1.00. The \ 
analyses of covariance are presented in Tables 3** and 35- 



3 
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Table 34. CTBS Reading Vocabulary Potest 
Analysis of Covartance 
Covarlats • Pretest 



Croup 


a 


Pretest Mean 


Posttest Mean 


As justed Mean ' 


Mean 
Difference 


Com per Ison 


23 


527 0 


556.1 


565.7 






38 


519.6 


564.0 


1 

563.5 


i 

2.2 


Context 


57 


550.3 


564.0 


555.4 


10 3 



Analysis of Covarlance 



Source „ 


S 5 & 


df 


M S 


r 


letwtrcn 


2416.3334 


2 


1209.1*67 


0.5262 


Within 


/•i »4J.72uu 


114 


2297.7520 




Intel 


264362. 05J4 


116 




» C .5923 



Table 3', CTBS K t „ini w <. abular y 1 o>a u «t 
Analyst * . f (.- \ ir fance 
CovarJdtt Prttrst 



Group 


n 


Cr* tc it Mean 


Pattest Mean 


Adjusted Mean 


Mr.nn 
Plf frrcmr 


ACF-OIney 


3« 


519.6 


564 0 


567.7 




Context 


56 


549.4 


0 

561.6 


$5919 


7.8 





Analysis n 


f <MV 


ir t.tneo 


> 


Source 


S S 


df - 


M S 




Betwrrn 


1 \R2 '.767 


) 


1382.6767 




Within 


229799.6000 


01 


2525.2726 




T\tal 


2311*2.4767 


9? 







p <.461) 
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Reading Comprehension . There was no significant difference between 
the ACE-Olney group and t the Comparison and ,Con 4 text groups on either 
analysis of covariahce; in each instance the F valine was less, than ] J30. 
The analyses are presented in Tables 36 and 37* 

Table 36. CTbs KeaJ irffStop: etu'islon Postteet / / 

* ' / 

' Analyst s of Cover lance . ✓ 

^ CovarJate « Preccac • x 

<3 



Croup 


n * 


Pretest Mfan 


Poet Lent Minn 


% Adjusted Mcun 


Mean 

DUUrvetis 


ACE-Olney 


38 


531.9 


344. 0 


537.4 




Conparlfton ^ 


23 


509.7 


524.2 


534.1 


0.6 


Context 


*57 


525.2. 


530.5 


52ft. 5 t 


9.1 



Analyale of Coverlance 



Source 


S S 


df ' 


M S 


F 


let ween 


2237.5406 


2 


Jllft.7703 


0.5532 


Within 


230549.2100 


114 


2022.3615 




Total 


t 2327 8 6 . 7506 


116 




»<.5767 



Table 37. CTlS Re.idtns Comprehension Poet test 
Analyala of Coverlance 
Coverlate * Pretest 



Group 


n 


* 

Pretest Mean 


Poettest Meen 


Adjusted Meen 


Mean 

Difference 


ACE-Olney 


3ft 


531.9 * 


544.0 


541.7 




Context 


56 


525.4 


53\o 


533.2 


^ ft. 5 ' 


- « 


1 r \ 1 





Analysis of Coverience* 



9 

ERIC. 



Source 


s s m 


df 


V * 
M S* \ 


P 


let ween 


4606. 48,8 


I 


. 1606 4 .4i5i \ 


0.ft263 


Within 


176926. 6 700 


91 


1944.2492 I 




Total 


171533. 150ft 


92 







• < .1651 
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Reading Total . There was no significant difference between groups 
on either analysis of covariance. In each case the ACE-Olney group 
performed about the same as its non-equivalent control groups. The 
analyses are presented In Tables 38 and 39. 

Table 38. CTBS Reading Total Poattekt 

Analysis of Covarlance * 
Covariate - Pretest 



Croup 


n 


Prete»£ Mean * 


Post test Mean 


Adjusted Mean 


Mean 

Difference 


ACE-Olney. 


38 


532.3 


351.2 


574.4 




Coapar lson 


*? 


515.5 


1 » 

534.9 


544.1 




Context 


57 


' 534.9 


544.7 


536.6 


8.e > 



Analysis of Covorlance 



1 

Sour Lt 


S^S 


df 


>■ N S 


" 

r 


fU t wee n 


1777.7368 


2 


888.893'* ' 


0.5680 


Within 


1/K41.7.2700 


114 


■ , . — 

. 1^565.0638 




Total 


180195.0568 


116 




p<.5683 



Table 39. CT8S RcJiUnf Total Poatteat 

Analysis of Covarlance * 
; Covariate -pretest - 



Croup - 


n < 


Pretest Mean 


Posttest Hean * 


Adjusted Hean 


Hean 
Difference 


Ace-01n«y **"* 


38 


/ 532.3 


551.2 


552.1 




Context 


56 


534.6 


544.5 


r 

543.* 


8.5 



Analyela of Coverlance 



- * 

Source 


s s 


df 


M 


r • 




M38.0098 




1 


1638.0098 ^ 


1.0575 


Within 


140952.9900 


91 


1548.9340 






Total 


73 


142^90.9998 { 

; 




'9^ 


; 1 — * *!* i 

• 




/ 


♦ 


» 








t 

P<.306e 


" - i 





a 



Arithmetic Computation . . Neither analysis fdentlf teij^jy slgnlf lea 

/ I ' 1 V 

difference between th^le ACEfOblney group and the non-tequlvalent contrdl 



groups , 



The analyses of covarlance are presented I 



Mlc 40. CTIS Arithmetic Computation Poattaat 
Anal ya la Df CovarUnce 
CDvarla|e - Preteet; 



n Tables kO and 



Croup 


a 


Prateat Mean 


r- 

fDittaat He an 


Ad J ua ted Maan 


» ■> 

Maan 
Difference 


ACE-Qlney 


36 


,496.3 


523.3 


529.1 • 




Context 


36 


302.0 


514.1 


314.9 


14.2 


CoaparlaDn 


22 


506.5 


513.6 


509. Z 


19.4 



An*lytH Df Covarlance 



Source 


/ 

S S 


it 


f 


9 

MS 


r 




•attain 


6434.8392 


2 


3217.419a 


0.9906 




Within ' 


357213.2400 


110 


3247.3931 


»<-374a 




TDtal ' 


36H48.0792 


112 







Table 41. Gll^ Arithmetic Onputatlon roetttat 
Analyala Df Cov.irlance 1 
Covariatc « Trrteat 



Group 


n 


"fttfteat Haan 


Post teat Hean 

" -z- — 


Adjusted' Mean 


Mean 

Difference 


ACK-Oluey 


36 


496. 3 


5?5,3 


527.1 Q 




Context 


56' 


502.0 


514. a 


313.0 


14.1 * 
















Analyats Df CDvarlanee 



ft 



Source 


S S 


if 


M S 


t 


fcrtwaen 


4362.2472 


J 


4342.2472 


i.34»9 


% Within 


216503.3100 * 


89 


* 3219.1392 


, V 


Total * 


•290145.4272 


90 





Arithmetic Concepts . The analysis of covarlance which Included 
the ACE-Olney, Comparisons and Context groups identified a significant 
difference b*tweei^jj roups • 4 When- Tukey Tests were performed on the 
differences betweten adjusted means, the ACE-Olney and Context groups 
were superior in performance to the Comparison group and equal to 
one another. The analysi^ of covarlance which included only the ACE- 
Olney and Context* groups confirmed this finding; there was no signi- 
ficant difference between the two groups. The analyses of~covar i ance 

m 

are presented/ in fables kZ arid ^3. 



Tabic 42. CTBS Arithmetic Concepts Poetteet 
Analyaie of Covarlance 
Covarlatc - Pretest 



\ 

Croup v- 


n 


Pretrn Mean 


Politest Wean 


Ad J ue ted M«nn 


, He an 
Different • 


ACE-Olney 


36 


5<ft M 


539. 3 X 


54a. 8 ! 




Confrxt 


5*. 


516.5 


544.7 


543.4 


3.4 


C*«pu r i to n 


22 


52J\2 


SOS. ft 


502.3 


44. J 



Analysts of Covarlance 



df 



M S i 



letvern 



318)0*2056 



15915.1028 



441736.7500 



no 



4015 7887 



V 473566^9556 



112 



/3 .9631 



( 



b <.02W 



Tukey Teat for Differences Between Adjusted Heane 



adjusted M»>an 
Dl Tf rrencc 


1 


2 


3 


44. i * 


'40.9 * 


2 

^ 1 


3.4 





Critical Value • 32.35 
P «M0 



Table -(IBS Arithmetic Connate Po»tte*t 
a Am>]ysi* of Cp\.irl.ince 
(Vv.irUte • TrctcBt 



Croup 


n 


Pretest Wean 


Postost Mean 


Adjusted Mean 


Mean 

Difference 


ACE-OUey 




506. 8 r 


539.3 






Concede 


56* 


516. S 


544. 7 


340.3 


3.4 



^ Analysis of Coverianc* 



{ 



Soiirc* 


s s 


df 


M S 


/ 

r 


Be twe^n 


, 251. nu 


1 


251.1312 


0.0603 


Vlrhln 


37O\89.41O0 


69 


4166.1732 




Total 


37)040.5412 


90 


p<\8066 



Arithmeti c Applications , The analysis of covarfance which 

•' * \ 

included the^ACE -01 ney, Comparisons, and Context groups revealed a 1 
significant difference between groups. Tjikey Tests on the difference 
bet/vefen_adJus^e4^Tteans identified the Context group as performing 
-si^nff Icantly laweV than the ACE-01'ney and Comparison groups which 
were equal to one another. The nalysls of covariance which included 

only the-A€£-4H/ney afid Context Groups aflso ider.L I f led th? significant 

r 

difference in posttest levels of performance. The analyses of xovar- 

1 • i 

lance are presented in Tables kk and *i5. 
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Tafcle 44, CTBS Arithmetic Application* Peetteet 
Analyeia *f Cov*rieace 

Cevnrleter • Prataet 



Group 


• 


Preteat Mm* 


Peetteet Keae 


Aejuetee 1 Mut 


Hui 

Differ mn 


1. ACE-Olney 


34 


490.9 


524.1 


317.7 




2. Comparison 


22 


471. • 


497.1 


304.7 


21 #0 


3. Context 


5*. 


314.2 


493.3 


412.9 


U.I 



Anelyele of CevarUncc 



Source 


S S 


4 I 




t 


Between 


43170.7402 


2 ' 


21933.3701 


3.9902 


Within 


404710.1900 


no 


3497.3434 

— "■ U' 




Total 


441510.9302 


112 












4 f 


f <T.0213 



Tukey Teat for Differences Between Aajuoteel Heans 



! jr 

Aijueted Nui 
Difference 


1 


2 


3 


44V 


23.1 


2 


21.0 





* Critical value - 37.13 
■ - .10 



V 



Tabic 45. CTHS Arithnatlc Application Posttcat 
Anolyiia qf Covnrlance 
Covnrlate • Preteat 



Croups* 


n 


Preteat Mean 


Voattaat Mean 


Mjuated Mean 


Mean 

Difference 


Acr.-OJncy 


36 


490.9 


324.1 


334.3 




Context 


34 


316.2 


493/3 


413.4 


41.9 



\ 



Analyala of Cover lance 



9 

ERIC 



Source 


S $ 


it 


N S 


r 


Between 


31230.7149 


I 


31230.7149 


10.4309 


Within 


428191,3600 


•9 


4119.0711 




Total 


» 400124.2749 1 


90 




a (.0016 



V7 
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Arithmetic Total . Neither analysis revealed any significant 

e 

difference between the ACE-Olney and Control groups. The analyses 
of covarlance are presented in Tables 46 and kl. 

* * 

i Table 46. CTtS Arithmetic Total Poo t tee t 1 

*>/ 

Analysis of Covarlance 
Cover late - Pre toe 



Groups 


n 


•A 

Pretest Mean 


Poet test Mean 


Adjusted Moan 


Mean * 
Difference 


ACE-Olney 


3* 


4S9.0 


523.4 


T! ■ 

521.2 




Context 


56 


505.6 


514.0 0 


509.5. 


19.7 


Comparison 


22 


' 50q.5 


302.2 


500.9 


21.3 



Analysis of Cover lance 



So urea 


S So 


4t 


N 5 




Between 


13214.2242 


2 


6607.1121 ' 


2.0311 


Within 


354224,6700 


no 


3220.2243 




Total 


367431,1942 


112 




P <.1334 



Tefcle 47. CT1S Arithmetic Total Postteet 
Analyele of Cover iance/ 
q Covarlata - Preheat 



Croupe 


n 


— j- 
Preteet Maen 


Poattaat Maan 


Asjuetee* Keen 


Haae 

Difference 


ACI-Olnay 


36 


469.0 


523.4 


526.4 




Centaxc 


56 


505.6 


514.0 


309.0 


19.4 



Analjr.lt of CovarLnc* 



ERIC 



lour c a 


t S 


if 


H 1 


t 


let vein 


1213.0324 


1 


1213.0324 


2.5256 


Wthle 


219424. 2600 * 


•9 






Total 


297637.1124 


90 




»<-US6 



The testing of the hypothesis that experimental students would 
demonsfrafe significantly greater gain in basic skills proficiency 
than control students failed to Identify any significant differences 
on any of the Reading subtests of the CTBS. Only two significant 
differences were identified on tk*e Arithmetic subtests of the CTBS: 4 
the ACE-Olney and Context groups w^re equal to one another aqd superior 
to the Comparison, group on the Arithmetic Ctfncepts~7ubteslTand the 
ACt-Olney group was superior to the Context group on * the Arithmetic 
Applications subtest. When the Comparison group was removed from 

the analyses, the only significant difference identified was the 

if 

superior proficiency of the ACE-Olney group on the Arithmetic Applica- 
tions subtest of the CTBS. • 

In general, the failure to rejett null hypotheses for all but 
two subtests of the CTBS (one if the Comparison groijp Is excluded) 
offers scant support for .concluding that experimental students gained 
significantly nfore In basic skills proficiency than did their control 
group counterparts. 

Hypothesis 3 . Students will gain significantly (p<.10) In career 
maturity . The hypothesis that students -would gain significantly in 
career maturity was tested by an examination of gains on scales of the 
Career Maturity Inventory (CMI>. The scales used were the Attitude 
Scale ^nd the Occupational Information and PlannTng competency" subtests. 
Correlated "t" tests were calculated for gains on. each of the measures; 
since a directionality of outcome was Incorporated in the hypothesis, 
one tail "t" test critical values were used. - The data used for the 

5*9 
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calculation of the "t" tests were the pretest and posttest +a* 

•* - 

scores for each student* 

The ACE students did not demonstrate any significant gains in 

* 

career maturity.' Rather, they showed losses on the Attitqde Scale 

# y> 

as well as on the Occupational Information and Planning subtests of 

"7- ' 

tKe CMI • The pretest and posttest levels and "t n tests ^or the ACE 
group are presented in Table 48. > 



/■ 



Table 41 
Career (taturlty Inventory Cains 
"ACE Croup 



Scale Fret ou t Poet tent Caiti t 

Auitude Scale 35.31 33.01 -2.22 -0.71 

Occupational Information 13.97 , 12.15 * -3.12 -1.50 

Planning 11.46 t.W -1.72 -1.11 

critical value: t » 1.3l,e - >10, 4( > 30 



f 

ERLC 



The ACE-Olney students demonstrated significant gains on all 
measures of the CHI. They showed significant gain onjjjKe Attitude 



Scale as well as on Occupational Information and Planning subtests 
of v the CHI. Gains , pretest and posttest levels, and "t" tests are 
presented lYi Table 49. 

Thus, the hypothesis that students would gain significantly 
in career maturity Is supported by the CMI performance of the ACE- 
Olney group and not. supported by the performance of the ACE group. 
The differential performance by the two "groups 'I s difficult to Inter- 
pret. Differential Career Guidance was provided to the two groups* 
\ 



\ 

\ 
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Taala 49 * 
Career Maturity Inventory Calm 
ACE-01*ay Croup 

Sc*Ua * Erataat fwttt ti Coin j t 

Attltua* Scolo 32.77 34.29 +1.52 lTl7* 

Occupational Information 14.15 15. U +1.31 * 1.5** 

Flawing 10.21 • 13.44 +3.23 3.H* 

critical value: t > 1.31, a - .10, it > 30 



the ACE group received one hour a week of Group Guidance and the ACE- 

«> . * 

Oiney group recleved two hours a week of Group Guidance. This* 

differential guidance does not explain the losses in career maturity 
^evidenced by the ACE group; one might expect iess gain but a ioss 
should not be anticipated since guidance was provided. One plausible 
explanation would be that career maturation is especially-facilitated; 
by the first year's participation in a Career Education Program; data 
are not available on the first year's growth of the ACE students and 
this explanation cannot be supported. This possible effect due to 
time of participation Jn the Career Education Program is a topic that 
requires further Investigation before any conclusions can be sub- 
stantiated. 

\ 

i 

0 

Hypothesis 4. Students will gain significantly (p<.10) more In 

. . . J 

career maturity than comparable students In a traditional school . * The 
hypothesis that experimental students would demonstrate significantly 
more gain In career maturity* than comparable students In a traditional 
* *o school was tested by comparing the* performance of the ACE-Olney, 

ERIC 

»Baprison and Context groups on the Attitude Scale and the Occupation* 

101 



Information and Planning, subtests of the Career Maturity Inventory 
(CHI). The comparisons were conducted by performing analyses q£» 
covariance on the posttest performance levels on the CHI measures; 
the entry level of the groups was made equfvalent through the use of 
the covariance procedure. Raw scores were the form of the data 
use'd. In all cases, the posttest performance score was the criterion f 
measure and the pretest* performance score was the covarjate. 

Two sets of analyses were conducted due to the previously 

$ \ 

identified non^equ I valence of the Comparison group: one set Included 
the ACE-Olney, Comparison and Context groups and v he otheY included 
only the ACE-Olney and Context gfeups. Each set Included an analysis 
for each of the CHI measures. 

» ^ *■ 

a 

Attitude Scale , . No significant difference between the ACE-Olney 
group and the ctmtr&l groups was identified when all three groups were 
included in the analysis. However, when the variability due to the 
Comparison group was eliminated fronTthe analysis, a significant dif- 
ference between the ACE-Olney and Context groups was revealed; "the 
ACE-Olney group was significantly superior to the Context group in 
attitudes measured by t fie Career Maturity Inventory. The results of 
the analyses of covariance were presented in Tables 50 and 51. 

Occupational Information Subtest . Significant between group 
differences were identified on both analyses of covariance. Jn the 
analysis of covariance which included all these groups, Tukey Tests on 
"the differences between adjusted means indicated that the ACE-Olney 

9 



V 



-/ 



Taala SO* CHI Att^ua*a Scala raattaat 
Analytic of Covariaaca 

Ccvarliti • Frataat 



Crouf t 




frttnc Km« 


Pnttiit Nmi 


Aal juataa H«u 


Nmi 

Dlf fur coca 


1. ACE-01»ty 


41 


32.77 * 


34.29 '' 


33.71 „ 




2. Ciitot 


30 


31. fO 


31.93 


31.12 


1.S9 ' 


3. Caapariaa a 


17 


30.41 


31.03 


31.73 


, i.H 



Aaalyala af Cavarlaaca 





a 


So urea 


S S 


4 f 


M 1 


P 


1+tveaa 


« 15.9304 


2 


42.H52 


1.1749 


Within 


1349.7414 


91 


" 2L.7334 




Total i 


•433. 47U 


93 







9 < .1447 



Taala 51. Qi\ Attitude Scala Paattaat 
AiMlynia of. Covarianca 

CavarJata • Prttaat 



Craaa 


■ 


Vrataat Maan 


Poattaat Haas 


Adjuataa Kaa* 


Haaa * 
Dif (areata 


1. Att-Olaay 


48 


32.77 


34.29 


34*05 




2. Caataxt 


3* 


31.90 


31.93 „ 


. 32.17 


l.M 



Aailyaia of Cavarlaaca 



Saurc • 


II 

— a» 


4 i 


H 1 


9 


Bat w«m 


44.9771 


1 


44.9771 


3.05H 


Withia 


1394.1975 , 


73 


21.2433 




Tatal 


1459. 1753 


74 








♦ 


* 




9 < .0044 



group was superior to the Comparison and Context groups which. In 
turn.were equal to one another. The analysis of covarlance which 
Included only the ACE-Qlney and Context groups >lso Indicated that the 
ACErOlney group was significantly better than the Context group In Its 
mastery of occupational Information. The analyses are presented In 
Tables 52 and 53. 
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4 Ta»lc 52. CM I Occupstlorul Infaraatian Poattcat 
Analyaia tf Covarianca 

Cavariata ■ Frataat 



Craup 



1. ACt-Olnay 



Frataat Hui 



H.15 



Hat test Maaa 



A4juata4 ttaaa 



13 .§5 



2. Ccwaarlaon 



17 



10.47 



9.97 



4.0i 



3. Caatext 



30 



10.47 



t.ao 



t 



•♦•9 



5.04 



Aaalyalaftaf Covarianca 













Saurc* 


f f 


4 t 


M S 


f , 

r 


Batvaa* K 


453.9454 


2 


224. f 127 


13.4747 


Wltkin 


1510.4117 


91 


14.3917 




Tatal 

* 


1944 [km 


93 ■ 


• 





• <.oooi 



Tukay Taat far Dlffaraoca lctwaaa Aijustaa?Maaaa 



JUjuataa* Kaaa 
Dlf f«r«»ct 


1 


2 


3 


3.04* 


0.91 » 


2 


4<M* 





Critical valua - 2.32 
a • .10 



Tafrla 33. CHI Occuaatioaal Iafarwatlaa taattaat 
Aaalyaia mt Cavarianca 
Cavariata - Irataajc •> 



Craua 


a 


Frataat Maaa 


Pa at taat Maaa 


A4juata4 Maaa 


Maaa 

•Iffaraaca 


1. ACI-Olaay 


41 


14^15 


15.44 


/ U.35 




2. CoattKt 


30 


10.47 


1,20 


t.31 


3.04 



Aaalyaia af Cavarlaaea 



lavrca 


f 1 


4 f 




t 


aaj^aan 


397 .0430 


1 


397.9439 


23.1214 


Vithia 


1 1252.5300 ' 


73 


U.70O4 




fatal 


1430.4719 


74 







* <.«001 
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Planning Subtest « Both analyses 'of cqvar lance indicated that . 

there was a significant' difference in the planning performances of 
» * • 

the groups. The analysis of covar lance which Included all three 

* • * 
groups produced an F Value of 27*9292. Tukey Tests performed on the, v 

-adjusted means JmUcated-that-^ihe^ ACE-01 ney gcoup-per formed sJ gn 1 f i can 1 1y_ 

better on planning tasks than did the Comparison and Context grdups; 

the Comparison and- Context groups were equal to ofte another. The 

analysis of covarjance which included only the ACE-Olney and Context 

groups also indicated that the ACE-Olney group was significantly % o 

better than the Context group In .plannlng^performance. 1 The analyses 

qf covariance are^ presented in Tables 5* and 55.^ » - * 



Table $4. 'Ml running reetteet 
Analytic e£ Cavcriance 

Cevariete » Pretest 



Creep 


a 

n 




PovMcct Menn 


4 

MJuHtc^ Mean 


Htm 


1. UWlrmy 


4« 


10.21 


13.44 


* 13. 39 




2. Ceanarieen 


17 


13.04 


7.11 


1.51 


3.12 


— a — — -i-i 
3. Centaet 


30 


5.70' 


*4.77 


7.13 


(.2* " 



t 



Analytic ef CoverUnce 



letirce 


ft 


4 f 


M * 


r 


Between 


• 74.31M 


2 


> 30. 1339 


27.9292 


Willi in 


1427.4442 


91 






total 


2303.9440 


93 
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ftfcey Teat for Difference ietwen Adjuetee 1 Ncana 



Mjunt«4 Meaa 






♦ 

• 


Difference 




2 




3 




1.31 . 


* Critical value-" 2.23 


* 


3.12* 







Am 



4 " 4 
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TaWS ». CMf PUo«ln| Paa<taat 
Analysla of Cpv«ri«»c« 

Cav»rUt« • frjuat 



• 


n 


frattst Hua ' 4 


Foattai* Jtoan 


Mjucti^ Haas 


Dlffaraaca 


1. ACI*01o«y 


. ,. # „. 
41 


10.21 


13.3* 


12.23 




2. Cantaxt 


30, 


$.70 


• 

7.13 


7.f7 . 






- r 






* 





AaalysU al Covarlanca % 

I 



Sourca 




d i 


M S 


F 


Batwaaa 


25 J. 2177 


1 


» 253.2177 


11. §710 , 


Within • 


<s - , 

M73.3330 




12.9771 






122S.4227 


7i, 




j 



a <,0OW 

♦ 

The three group analysis of cava fiance showed that the ACE- 
Olney group -gained significantly more chan did the. Comparison and Con* 
text groups In the Career Maturity Inventory Occupational Information 
#nd Planning competency subtests* The three group analysis revealed 
no differences on the AttJtude Scale. When the Comparison group 
was el Imlnated from the analyses due* to Its doubtful comparabl 1 1 ty t , 
the ACE-Olney group showed more gain than. the Context grouj^ on the 
Attitude Scale as well as on the Occupational Information and Planning 
competency subtests. * , , \ y 

The results of th« analyses clearly Indicate that the ACE-Olney 

group gained significantly more In career Maturity than the control , 

* • . . » 

groups as Indicated by the competency subtests of the CMlr The two > 

c 

group analysts Indicates that the ACE-Olney group also gained significantly 

* A 

q more In attitudes than the Context group. 
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Thus, there is strong support for the conclusion that the ACE- — 
Olrtey students. gained significantly more In all measur^l aspects of 
iXtwturity than did comparable students Ip a traditional hi'gh 



career 
school 



Hypothesis 5.« Students .will evidence a s Ignlf Icahtlv* (p.'<1(0 "tore 
positive attitude toward school thao students In a traditi onal school. 
The hypothesis tttat experimental students would evidence a significantly 
more Dpsltlve attitude toward school than their control counterparts f 

was tested by comparing .the performance leyejs of the ACE*Ol*>$VVCom- 

• * * * ^V 7 ^ 

parlson and Context groups o<( the Assessment of Student Attitudes^oale 

* *' ' * * 

(ASA) . The compar i sons ^ere conducted by perfoYmlnganalyses of . # , ^ 

* « • * 

covarlance on the post'test -performance levels on the subtests and fver-^ 
all mean of the ASA. Analyses were thus performed for Attitude Toward 
Education In, General, AttI tude; Toward School Curriculum, Attl tude- r __ 
Toward School Resources, Attitude Toward School Counseling, and Overall. 
Attitude Toward Learning Environments. Raw scores were used ^or each, 
of the subscales and overall means were used for th^ Overall Attitude. 
Toward Learning Environments- For all analyses,, the posttest level pt 
the attitude scale was* us.ed as the criterion measure and tne. pretest ^ 
level was used as the covarlate. a/ 

Due to the previously Identified lion -equivalence of t^'^ompar I son- 
group, two sets of analyses were conducted. One Included* the ACE^Olney.- 
Comparlson, and Context groups and the other Included. qfely/thq ACE-Oljie*, 
and Context groups. Each set Included an analysls»-for each. 'of tfha ASA 



measures . 
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' Attitude TdWard EducatlorTln General . Neither analysis of covarlance-. 
revealed any slgnl f Ican^ dlfferericcs between groups In their Attitude ' . 
Toward Education In General. The analyses of covarlance are presented 
In Tables 56 and 57. 

K *able 54. ASA Attitude Toward Education' In General * 
^0 - Analyifs of Covarlance 

. Covurlatc ■ Pretest jj* 



Group 


n 


Fretcet Mean 


Foittfst Mean 


Adjusted NMD 


Me*e * 

Differences 
■ 


1. ACE-Olnty. 


32 


^43.6 


. 334.3 * 


3)1.3 




2. Cooper leon 


22 


333.0 _ \ 


344.2 


344.2 


* 3.3 


3. Context 


56 


v 337.2 


\ 333.7 


"336 .2 


14vl 



Analysis of Covarlance 



Sqyree 


S S 


d f 


. ' MS 


P 


Betvuen 


4751.* 501 


2 


2379. V5* 


0.4333 


Within ( 


33 3123. A 500 


106 


3224.7414 - 




Total 


. i3#5l5. 400ft 


]oa 
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Telle 37. ASA Attitude Toward Educe t lee ta ^CiumI 
Aaeiyele of Cover lance 

♦ ' . 

Coverlet* ■ Pretaat 



Craue 


a 


Pretaat Keen* 


Peett'et Keen 


' Ad jus tod Keen 


• Moee 

^Difference 


1. ACI-Olney 


. 3? 


433.4 «• 


334.3 


«2.7 




2. Context 


So 


137.2 


■ }«e? 


MM. . - 





Analyst* of Cover lenco 



« Seurce 


1 S 


e f 


M S 


P 


let ween 


4411.2247 


T 

1 


4411.2247 


0.1322 


Vlthle 


470931.3300 


15 


3340.4117 




Total 


4 7 3349 . 7347 


14 







. Attitude Toward School Curriculum . The analysts of covar lance 
which included all thrlse treatment groups did not reveal any significant 
differences between the groups In the attitude toward school 
curriculum. However, when the variance due <to the CompaWson group 
was eliminated from the analysis^ the a n a lysis of cov a r fohce Indicated 
that. the ACE-01ney group had a sf gnl f Icantly more positive aftltude 
toward school curriculum, than did the Context grojjp. The analyses 



0f covarlance are presented in Tables'58 and 59. \X 
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T«H« 54. ASA Attitude Tovord School Curricula* 



Aaoly^i of Cmrltici 
Cmrliti • Trotoot 



Group 


n 


Fro toot Mtoo 


Poottott Hooo 


Adjuotod Hoon 


Noon 

Difference 


I. Ad-Olnoy 


32 


i 

343.0 


349.4 


377.1 k 




2. Coooarlton 


22 


370.0 


341.2 


344 ^ 


13.3 


3. Ctntoxt 


34 


372.1 


331.4 


347.. 1 


30.7 



Anolyolo of CovoriWo 



Icotco 



Ittvooo 



tflthlo 



I s 



17133.3441 



311322.0000 



137.3441 



4 f 



45 



4ft 



M S 



~* 1743*. 3441 



ftOlS.fBfr 



.4 f 



2.9449 



Toolo 39. 



4*A Attitude Tewd School Cutrlculua 
* S K 

Aoolyals of Covoriooco 

m «- 

Coverloto - Pretett 



* < .0444 



Crcuo 


0 


Pretest Mom 


Poettest Mton 


N 

Ad jutted Hom 


( , . n 
Moan 

biff or once 


1. ACI-Olney 


32 


345.0 


349.4 






2. Coot out 


34 


372.1 


351.4 


345.4 


10.0 











Aaolyslo of Covoriooco 



f 

Source 


S S 




. M S 


P 


Between. 


19443.3311 


2 


9741.4459 


1.7114 


VltoU 


^403344.7700 


104 


* 5492.1203 




Totol 


422444.1014 


104 
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Attitude Towafrd School Resources , Each analysis of covar lance 

i *9 — 

Ind Icated sthat th4re waT~e? significant between groups difference In 
attitude toward School resources. Tukey Tests performed on adjusted 
means resul ting from the three group analysis indicated that the ACE- 
Olney group had a signficantly more positive attitude toward school 
resources than did the Comparison and Context groups; the Comparison 
and Context groups were equal to one another. The analysis of 
covariance which included only the ACE-Olney and Context groups also 
Indicated that the ACE-Olney group had a significantly more positive 
attitude toward schodl resources than did the Context group. The 
analyses of covarimce are presented in Tables 60 and 61. 



Tefcle 60. ASA Attitude Toward School leeeurcee 

<a> 

Anal ya la of Covarlance 
Covarlate * Prateat 



Croup 


n 


Pretaat Ma an 


Poattett Mean 


Adjusted Hcnn 


Mean 

Difference 


I. ACr-Olney 


si 


319.1 


310.3 


315.4 




2. Comparison 


h 


332.0 


345. 5 


344.2 


41.4 


3. Context 


5ft 


337 .3 


324.9 


320.1 


44.1 



Aaalyala af Covarlance 



Sour c a 


S t 


a* I 


M S 


t 


Between 


•4114. 0730 


2 


42057.0345 




'Jlcnln 


39353a. MOO 


104 


3712.4102 




Total 


477450.7530 


101 







Tukey Taat for Difference l«tvrcn Adjuatcd MaaM 



a < .0001 



Ad jut tad Mean 
Dlfftreace 



44 V 



41.4" 



23.4 



Critical value • 31.70 
a • .10 



J 1 th 



***** 
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T«Ut II. ASA Attltuea Toward ScWl Rttourcci 
Aailyiie ef Coverlance 

CfvirUct - Pret««t 



Croupe 


n 


treteet Keen 


Foetteet Nun 


Adjusted Mean 


Nun 
Difference 


ACK-Olnty 


32 


319. t 


3*0.3 


313.9 




Context 


56 


6 337.5 


324.9 


321.3 


■2.4 



Aaelyele of Coverlence - 



Source 


S S 


dt 


N S 


f 


Betveen 


7Slt6<. 6852 


1 


7S166.«S$2 


19.1107 


Within 


335313.2900 


S3 


3945. tlft 




Toul 


41354J.9732 


•ft 




»<.0001 



Attitude Toward School Counseling . The three group analysis of 
covarlance revealed no significant between group differences in 
attitude toward school counseling. When the Comparison group was 
removed from the analysis, the two group analysis of covarlance Indl 
cated that the ACE-Olney group had a significantly more positive 
attitude toward counseling than did the Context group. The analyses 
of covarlance are presented in Tables 62 and 63. 

in 
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Tabic 62. ASA Attitude Toward School Counseling 
Anelyeie of Covorlence 

Covorlete - rrctcat 



Croupe 


n 


frctcet Mean 


Poettcet Neen 


Adjuetcd 


Meen 


Neen 

Difference 


ACE-Olney 


32 


282.5 


340 .0 


340.4 




Coaparleon 


22 


273.6 


300.0 


304.0 


36.4 


Context 


56 


294.6 


307.5 


303.1 


17. 3 








Anelyeie of Cover lance 






r 




Source 


S S 


df 


H S 


P 




Ictween 


31193.7978 


2 


15596.8989 


1.987ft 




Within 


•31779.9900 


106 


7846.9811 








Totel 


862973.7171 


108 ' 






p<.1421 






Table 63. ASA Attitude Towerd School 


Couneellnit 












Analyels of Coverlnncc 
















Cover let e 


- Freteet 










Groupa 


n 


Preteet Neen 


Foetteet Mean 


Adjueted Mean 


Mean 
Difference 


ACE-Olney 


32 


282.5 


340.0 


342.8 


i' 


Context 


50 


* 294.6 


307.5 


305.1 


37.7 






Anelyeie 


of Coverlence 










Source 


S S 


df 


m a 






F 




Ictween 




23361. 49A? 


1 


213M .49*12 


1. 


191/ 




Within 


710772.5400 


85 


8362.0299 








Totel 


734134,0382 


86 




• <.0691 
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Qvgrajj Attitude toward Learning Environments , Both analyses of 
covariance Indicated that there was a significant between group 
difference In overall attitude toward learning environments. Tukey 
Tests performed on the adjusted means of the three group analysis 
indicated that the ACE-Olney group had a significantly more positive 
attitude toward learning environments than the Context group didf the 
Comparison group was midway between the two and did not differ signi- 
ficantly from either the ACE-Olney group or the Context group. The 
analyses of covariance are presented in Tables Sk and 65. 

* Table 44. ASA Overall Attitude Toward Learning Environment a 

Am lye la of Cover lance 

Covarlett • Prtteet 



Croup • 


n ' 


Fro teat Haan 


Poettaat Haan 


Actuated Keen 


Keen 

Difference 


l>A€E-3>lncy 


32 


328.2 


341. t 


343.0 




2. Caspar laon 


22 


32e\* 


342.9 


343.1 


19.2 


3. Context 


56 


334. t 


330,3 


321.2 


34. • 



Analyaie 



of tbva 



Source 


S S ^ 


df 


H • 


r 


letveen 


24IM.S504 


2 


12404 . 2753 


3.1954 


Within 


4U710.C3CO 


L04 


3M4.0549 




Total 


434511. 5406 


Loa 


» ^.0450 



Tukey Teet for Difference letveen Adjueted Heene 



Adjueted Haan 
Difference 

■->h ■ 


1 


2 


3 




IS. 6 


2 


19.2 





* Critical velui • 32.42 
• • .10 



Table 45. Overall Attitude Toward Learning Environment* 
Analysis of Covarlance 
Cov*riat* ■ Prttcat 



Groups 


a 


Pretest Mean 


Posttest Mean 


Adjusted Mean 


Naan 
Differ once 


ACE-Olney 


32 


321.2 


361.1 


* 3*3.2 




Context 


56 


334.6 


330.3 


328 .t 


34.4 



! 

Analysis of Covarlance 



Source 


S S 


•4 


8 1 , 


=Ju= 


Between 


24044.0750 


i 


24044.0750 


5.747^ 


VI thin 


354UI.22O0 


ts 


4173.1413 




Total 


379034. 29S0 


16 
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The analyses of the ASA which Included the ACE-Olney, Comparison, 
and Context groups Indicated that the ACE-Olney group had a\ significantly 
more positive attitude toward school resources than did. the Comparison 
and Context groups and a significantly more pos I tlve overall attitude 
toward learning environments than did the Context group. 

When the Comparison group was omitted from the analyses of covarlance 
the ASA Indicated that the ACE-Olney group had a significantly more 
positive attitude than did the Context group toward school curriculum, 
school resources, and school counseling as well as a significantly 
fore positive overall attitude toward learning environments. Ohly 
on one. subtest of the ASA did the ACE-Olney group not show a significantly 
more positive attitude than the Context group: Attitude Toward Educa- 
t Ion In General . 

Hypothesis 6. Students will gain significantly (p<>10) In career 
knowledge over the course of cluster experiences . The hypothesis that 

11 A . 
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experimental students would gain significantly In career fcnow f e dg e ov er 

0 

the course of their cluster experiences was tested by an examination 
of their gains on the competency subtests of the CMI . The Occupational 
Information subtest measures knowledge of characteristics of occupa- 
tions and the Planning subtest requires knowledge of sequences of 
factors related to occupations. Both subtests thus requi re careet 
knowledge; the Planning subtest requires additional ordering of that 
knowledge. 

Correlated "t M tests were calculated for the ACE and ACE-Olney 
group on each competency test. One tail "t" test values were used 
since directionality of outcome was included in the statement of the 
hypothesis. The results of the correlated M t M tests were reported in 
the testing of Hypothesis 3> The ACE students showed no significant 
gains in career knowledge; rather, they showed losses on both measures. 
The ACE-Olney students did show significant gains in career knowledge 
as indicated by both the Occupational Information and Planning 
subtests of the CMI. r 

Thus, there is partial support for the conclusion that 
students will gain significantly in career knowledge over the course 
of cluster experiences. First year participants showed significant 
gains. The losses exhibited by second year students cannot" be fully 
interpreted since first year data, are not available on these measures. 

A 

i 

t 

Pther Effects Hypotheses 

Hypothesis 1 « Employers will be able to provide learning exper- 
iences sufficient to meet student needs and Interests ,, Th^s hypothesis 
is to be tested using three adequacy ratios each yielding a percentage 



statistic. They are: 

1. the total number of student places available In operating 
employer learning experiences divided by the total number 
of student places necessitated by program requl remervts; 
» (Adequacy by Need) 4 

* 2. the number of student first preference places aval I able 

In each cluster divided by the nqmber of first preferences 9% 

expressed for each cluster - summed across cluster 
(Adequacy by Interest); and 

3. the total number of student first preferences actually 
assigned divided by the total number of first preferences 
which could have been assigned under Ideal scheduling con- 
ditions (Scheduling Efficiency). d 

The first statistic tests gross meeting of needs. The second corrects 

for interests and yields the percentage of correct student assignments . 

possible given expressed Interests and actual places, available in 

each cluster. The third expresses the percentage of correct assign* 

ments actually made in light of expressed interest. Differences 

between 1 and 2 reflect the fH between the distribution of places 

t 

available In various clusters and the places desire^ in those cluster's* 
Differences between 2 and 3 reflect scheduling M sl Ippage": cluster 
assignments which could have been made given maximum operational 
efficiency vs. those actually made. Thus, each statistic measures a 
different aspect of the adequacy of employer learning act Ivl ties avail- 
able. No percentage level was preestabl Ished as acceptable. Obviously 
100% Is good; less than 1001 Is not as good. 



Career Exploration . Program requirements Indicate a minimum 
of 6 career explorations for each student during his Academy tenure* 
Gluen the student population, this requirement necessitated a total 
capacity of 125, 92 and 84 places In career exploration for the Fifth, 



* j I 

Sixth and Seventh Quarters respectively. The need declines as 
continuing students complete the requirement; lit increases as 
new students are added. The«c1usters of employers offering courses 
had total student capacities of 160, 172 and 152 for the Fifth, 
Sixth and Seventh Quarters respectively. This yields Adequacy by, 
Need figures of 128*, 1 87* and l8l%. Adequacy by Need thus exceeded 
th^ requirements. ' ' « 

In the Sixth Quarter the ffrst choices f^r cluster assignment 
were known (completed Student Needs^nd Interests forms) for kk 
students'. Of these, 40 could have been given jthelr preference - 
given the places avaltubte In the clusters of their choice. In the 
Seventh Quarter the first choices were Rhown for 75 students. Of 
these, 62 could have been given their preference under Ideal schedul- 
ing conditions. No data were available for the Fifth Quarter. The 
Adequacy by Interest figures were thus 91* and 83%. 

In actual scheduling during the Sixth. Quarter 32 students were 
given their first choice. During, the Seventh Quarter 5*» students 
actually received their* first choice. This yields Scheduling 
Efficiency quotients of 80* an$J 89* respectively for the two quarters 
Overall, these three statistics support the ability of the 
ecru I ted employee pool to meet the needs and Interests of students. 



Room {or Improvement In the fit of available places to student 
° Interest S^bv cluster and efficiency In scheduling Is Indicated. It 
should be noteckthat only student first cholte was employed In these 
analyses; second, third or fourth choice was obtained In almost all 
assignments. It should aHo be noted that a* substantial number of 



4 

students either did not express preferences, or those preferences 
were not recorded. • 

Career Specialization . In this area only Adequacy ttf Need Is 
used as a measure since a predefined pool of employer resources did 
not exist. It was attempted to recruit an employej to meet each, 
expressed, student need and .Interest. Adequacy by Need thus measures 
the ability of the program staff to identify and secure employer 

resources to fulfill student requests. 

• • • * 

In the Sixth Quarter 2$ students requested career specializations. 

» * 

This defined the extent of both student neods and Interests. Of 
s 

these, 19 were successfully placed in the specialization of their choice. 

* 

In the Seventh Quarter out of the 25 students who requested special iza 
ttons, only 10 could be placed. No comparable data were available 
for Fifth Quarter. The resultant Adequacy by Need quotients were 76* 
and h0%. t 

Overall, these statistios do not lend strong support Tor the J 
ability of employers to meet- the needs and interests of students In 
career specializations. This area requires extensive effort in future 

i 

project years. x 

• « * \ 4 i 

Hypothesis 2. Employers will cvhlehce a positive attitude and 
commitment regarding the program . It was intended to test this hypothest 
with results from the Employer Questionnaire. This instrument was not 
successful in either form, or procedure. Only a small number of 
employers (21) completed the instrument and the validity of obtained 

4* V 



results was quest ionned due to tjt\e generally negative reaction of 

employers to the prbiess\ /Thus, this hypothesis cannot be formally * 

tested. Only gross Indications can be mentioned. * ' . 

«» * * ~ * * • ■ .... 

Results from the Employer Questionnaire* suggested a general 

satisfaction with, the prpgram and the students Involved. A sizeable 

minority of employers* however, did Indicate concern $bout the level 
* ^ * 

of interest exhibited by their student,sl Must^fenployePs Responding 

were concerned about feedback from the projject. Feedback on the plan- 
ning and Implementation of .their learning activities; and particularly 
data on their effectiveness with students, was seen as lacking; A 

fairly high level of^emplQyer commitment Is suggested by' the fact 

. H * " • v 
I that rewer"lFarT20t "wl thdrevTf rbnTtfil T-program^dar Inrg FY^ig7^ — » 

i 

Hypothesis 3^ Parents will evidence a positive attitude and 

§ 

commitment regarding the program. In order to gather .data relevant ^ 
• ■ «* * 

% to this hypothesis two sets of Items "were designed for the Parent 

Opinion Survey. This Instrument was administered to the parents of 

studentS^ttendlng the Academy for Career Education; full results o 

are present^ tn Appendix C. The-ltem sets selected for this hy|>oth^sls 

were "Attitude Toward the Program In'General 11 and A'Benef I ts of the 

■ • 

% Prog ramVv—fach^l tern had~a~ response scale* from 1 (negative) to 5 
^ (positive). The specific Items and mean responses appear below.' 
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Mil «f 

Attitude Toward tht Frogren In GtMnt 



I tana 


Ha«n 


1. Mow wall doae tha Caraar Educe tie* Pre gran 
ceapera everall wt*th th« p*»t echoel 
exeerlentee of y«ur daughter/eon? 


4.47 


2. Zf yeu aed It fa do ertr again, vauld 
yeu want your eon/daughter to partici- 
pate in tha Caraar Educatlen Pronraa? 


4.43 


3. Nav wall 4a» yeu think yaur aan ar " r 

dauffcter llkaa tha Caraar Education Pvegrooi 
ctanrt^ with Mat achool axMrloncao? 


4, St 


Averoftoo 




Taole n7 
lanaflta af tha Pragran 




Haan 


t. Rave yeu race Wad enough Indorsation 
aeeut yaur oon ar daughter 'a progreae 
In the Caraar Education ProgroaT 


4.00 


7. la cooper lean with ragular echoolo ha* 
nuch opportunity 414 tha Caraar Educa- 
tlan Pregroa previde yaur 4a ugh tar ar aaa far 
laamlnf aoeut occupatlona? 


4.71 


1. What offoct. If any, haa tha Caraar 

Education Progran ha4 on helping yaur aan 
or* deufehter for* enrcar elan*? 


4.41 * 


• 9. In conparlnon with ragular achoola how 
nuch eppertunity^dld tha Car ear g4ueatlaa 
Fragran provide yaur aon or daughter for 
Canaral learning? 


4.31 


10. la conperleon with pa at aspcrlencto In 
ragular achoalo how nvtlvettd 1a your 
daughter ar aon to Warn In tha Career 
!4ucatlon Frogren? " » 


4.49 


11. Haw would you rata tha aeproachea to 
learning uaa4 In thla Caraar Education 
Proiran? 


4.43 


Aver ana 


4.44 



s 



As can be seen from these results parents expressed an extremely 
high opinion of the Academy program with most responses nearlng the 
positive extreme. Since the* rate of return on the Parent Opinion Sur 
vey was approximately 501, an unknown sample bias wi| Incurred. 
Existing data, however, argue strongly that parental support for the 
program Is high. 
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♦ * 

* Hypothesis *». Institutional relationships will be established 

« 

to enable the, conduct of the program . In addition to the commercial 
and Industrial firms necessary to provide specific learning resources' 
as discussed above, two principal Integrating sets of Institutional ' 
relationships and structures need to be established* In order to con% 
duct the Academy for Career Education Program.. The first Is with the 

♦ 

public school district. It Is thls^set of relationships which 'allows 
the program to operate with public school students as a population and 
determines the future of the program outside *of the experimental 
context. The second Is with the Chaipber of Commerce. It Is this set 
of relationships which facilitates the participation of community 
resources and . In combination with the -school' district Interactions, 

0 

seeks to accomplish the goal of uniting business and education In common 

* 

enterprise. 

The testing of this hypothesis U not amenable to any statistical 
*> * » i 
treatment. The relationships either exist or they do not. The Issues 

6f quality an< longevity lay outside of the scope of this report. It- 
can be stated that' the desired relationships were established In a form 
which seemed to meet the objectives. » , 

"The Philadelphia School District has become an Increasingly Integral 
partner 'in the Academy for Career Education. In FY 197** a cooperative 
program was established wherein Academy students received much of their 
educational program through vthe Academy on a released - tlme and after- 
class basfls, while having public school resources open to them.' Basic 
state credit requirements were successfully met by Academy learning 
actlvltjes. Student public school scheduleSyWere substantially adjusted 
to accommodate Academy programmatic needs. During FY 1975 these rela- 
' >V 
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ships are expected to continue their developmental pattern with an 
Integration of Academy and public school facilities and ^tifC. ' 

The Greater Philadelphia Chamber of Commerce has functioned well 
in identifying community economic resources and Initiating c&ntact for 
the Academy. Thl<$ successful funct Ion and the relationships developed 

within Its context have enhanced the rol$ of the Chamber as a viable 

* 

Intermediary .between the economic and educational communities. 

TheSe results strongly support the contention that institutional' 
relationships which enable cpnduct of* the program can be established, 
fhey further Suggest that, beyond facilitating the demon st rat tonal 
Academy program, a rep 1 1 cable and larger scal^ expert ence- based career 
educational program Is feasible in the public schools. s 

- . /, K» 

* - r , ^ 

% Hypothesis 5> It will be demonstrated that the program can be 

operated on a feasible cost. basis . In the "FY 1 97^ Operating Plans*" 1 * a 

« 

goal for the Career Education Program was to reduce the FY 1973 per 
pupjl cost of instructional services 30 percent; the targeted cost 
^eV student for FY 197* was $3,309. 36- This targeted cost Is accepted 
as the definition of a feasible cost. 

Costs for FY 137* *re based orr total expenditures for the core 
components of the program (Career Development; Career Guidance, and" 

O i 

Basic Skills). A fourth component (Supplementary Activities) has 
been omitted from the cost analysis since It will n& longer be. provided 
by the Career Education Program and since It was pffered only to senior 
students. The costs of program administration have been prorated to 



the costs of 'each core component. 



The total cost 6f providing the RBS Career, Eduction Program In 
FY 1 974 was $29*»,*»12. • Based on an average enrollment of 13*r students 
the cost per student for FY 1 97 k was $2,197. 10; ~th1s figure Is 
$1,112.26 below the definition of a feasible cost per student. Thus, 
FY 1974 results support the hypothesis that the program can be 
operated on a feasible cost basis. 

Table 68 .presents cost Information regarding total expenditures, 
cost per hour scheduled, and the cost per student. 

: . / 
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4.44 
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all 


1294.412 


54.144.3 
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134 
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It was reasoned that the post-graduation effects of the program were 
the most significant ones and a start should be made In the longitudinal 
investigation of effects. The results of this Initial Investigation 
are briefly reported here because they are pertinent to a summatlve 
evaluation of the program and because they suggest direction for future 
study. 

Three student groups were selected for the longitudinal study. 
The Experimental group consisted of 67 students who had graduated from 
the Academy during FY 1974. The Control group consisted of Ik 
students from the Context group who had graduated from Olney Hlqh 
School during FY 1974. The Non-Graduate Group consisted of 21 students 
who had dropped out of the Academy program during FY 1974. 

These students were administered a brief survey by telephone 
near the end of the summer. Approximately 901 of the students. in each 
group could be contacted for response. The results are thus fairly 
complete. 

The key survey Items with the responses obtained are presented 
below: * 



1. What Is. or will be your present education or career situation? 



Ca tego ry_____ — 

^ Students. 


Experimental 


Control 


.Non-1 


Srads 


1 






* 


f 




Post Secondary School 


31 


53 




38 


0 


0.0 


Full -Time Employment 


7 


12 


4 


19 


5 


26.3 


Part-Time Employment 


2 


3 


1 


5 


1 


5.3 


MM Itary Service 


6 


10 


1 


5 


0 


0.0 


Full -Time Homemaker 


2 


3 


0 


0 


0 


0.0 


Still Planning* 


11 


19 


7 




13 


68.4 


Totals 


59 


100 


21 


lob 


19 


lOO.G 



* ,0f the non-graduates in this category eight were enrolled In 
their original high school and five were unemployed, at the 
time this questionnaire was given. 
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2. Has anyone assodiated with the Academy/01 ney been of help 
to you In getting a job or getting Into a training/education 
program? 



Response^ — — ' 
„ — r- — * " Students 


Expermental 


Control 


Non-Grads 


# 


% 


I 




# 




Yes 


. 27 


46 


■ 6 


29 


8 


42.0 


No s 


28 


*7 


5 


2l» 


9 


47*5 


No Response* 


1» 


7 


10 




2 


10.5 


Totals 


$9 


100. 


■ 21 


-166 - 


19 


100-0 



* The control students irv this category could not decide 
whether guidance helped them or not. The experimental 
and non-graduate students in this category did not 
respond. 



3. Did your experiences in school help you make a decision 
about what career you f Would like to follow?. 



Response^ — - ^ 
- — *" Students 


Exper 


i mental 


Control 


Non-Grads 


# 


1 


If 




U % 


Yes 


38 


64 


• 11 


53 


Poes not 


No 


18 


31 


10 


. 47 


apply 


No Response 


3 


5 


0 


0 




Totals * 


59 


100 


.-21 


100 





As can be seen from these results, Academy graduates were more 
•one toward further education and were more firm In their planning 
than were control' graduates . Both Academy graduates and non-graduate^ 
reported substantially more, placement help from the Academy than 
control graduates reported for their school. A greater percentage of 
AcLdemy graduates felt that their school experience was helpful In 
career decision-making. 

These results support the efficacy of the program in faci 1 1 tatlng 
career planning. They also suggest a fostering of desire for post 
secondary education In preparation for a career. This initial 
Investigation may serve to highlight the need for longitudinal study 
of progranj effects. Equivalent, control groups are needed for drawing 

1 7.fi 



definitive conclusions. Time series data are required for documenting 
development. Follow-up studies should assume Increasing priority as 
the need for summatlve evaluation In program dissemination increases. 

V. SUMMARY -AND RECOMMENDATIONS 

Overview \ 

* 

Research for Better Schools, Inc. (RBS) Is responsible for 
developing, operating and testing a prototype of experience-based 
career education (EBCE) . This program has been operational ! zed 
In Philadelphia as the Academy for Career Education (ACE). The 
Academy operated during FY 197*» as a licensed private academic school 
with senior students receiving their total educational program from 
the Academy. All senior* were In the program for thelf second, 
year. All juniors and sophomores were In the program for their first 
year during FY 197*. Their curriculum differed somewhat as a result 
of an Increasing Integration of the Academy with the Philadelphia 
Public Schools. Juniors end sophomores participated In the "core" 
Academy program, while taking some courses (foreign languages, driver 
training, physical education, etc.) at their sending school. Students 
thus fell Into two distinct groups: seniors who attended the Academy 
courses only, and juniors and sophomores who attended some public 
school courses In addition to Academy offerings. All students partlcl 
pated In the M core Instructional component 11 which consisted of three 
major subdivisions: Career Development, Career Guidance and Basic 
Skills. This core constituted the bulk of each student's academic 
' program with at leas* \k hours of Instruction per week. It Is this 
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core program which forms the substance of the present report* The . „ 

r, ^ 

other instructional component , Supplementary Programs, was availabfe 

■f 

only to seniors and has been discontinued as movement is made toward 

§ 

having the pub 1 \c ti schools adopt an experience-based career education 

( 

program. 

o 

~"The first subdivision within the instruct ionul core was Coreer 
Development. This consisted of Career Exploration and Career Special iza- 
tipn activities for students. In Career Explorat ion. students experienced 
yroup programs (selected by them out of a large number of possibl I ties.) 
at community and industrial, si tes provided by employees of the various 
businesses, .unions and agencies involved. Students participated in 
these first-hand activities in order to learn. about the jeconomic 
community, to test their own vocational interest, and to obtain information 
for their career planning. In Career Specialization students selected 
a specific career area and investigated it in depth. These experiences » 
were highly individualized and required extensive interaction at a 
single community site. 

The second Subdivision within the instructional core was Career 
Guidance. This consisted^ princ i pal ly of small group guidance 
sessions which met each week. These sessions focused on life stills, 
academic motivation, integration %A Academy activities, self explora- 
tion and career planning. Can»«»r Guidance also l»w luded indiviilu.il 
counseling o^ students. 

The third subdivision within the instructional core was Basic 

Skills. Students were scheduled for activities In an Individualized 

v 5 

Learning Center several times each week. These activities focused 
on development" In Communication Skills and Mathematics and utilized 
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a variety of Individualized instructional resources. The Individualized 
Learning for Adults approach was the primary learning system. 

These subdivisions combined to form a core of Instruct lonal 
activities which were characteristically Individualized and responsive 
to student needs. The Academy program was designed to maximize student 
development and choice within an operational structure which could 
serve a large number of students at feasible costs. 

During FY 1974, over 150 students participated In the Academy for 
Career Education program. Of these, 67 graduated this year after 
spending two years at the Academy. The remainder were juniors and soph- 
omores who will return next year. The Academy student body seemed to 
be representative of an urban school population. 

The evaluation of the Academy for Career Education program has bfeen 
conducted by an Internal evaluation staff. Evaluation activity 
were of five principal types: formative evaluation, summatlve evaluation, 
data systems development, Instrument development and cooperative 
research.' The latter two activities occurred within the context of an 
active and productive formal Interchange among the four EBCE project 
evaluation staffs and the sponsoring agency (NIE) evaluation officer. 
Data systems development was undertaken to provide accurate Information 
In program operation management and evaluation. The formative and 
summatlve activities formed the major occupation of the evaluation staff. 
These are discussed In depth below as they relate to evaluation 
recommendat i ons . 
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Formatlve Evaluation * 

Formative Evaluation was accomplished through the administration 
of questionnaires to project participants, the Inspection of project 
records apd Interviews with staff. The formative structure was based 
on assessing key tasks within' each major program subdivision. All 
pertinent Information available was applied to the evaluation of these 
tasks. Results and recommendations are discussed below for each sub- 
' division. 

Career Development . The process of Identifying and recruiting 
employers using both the Greater Philadelphia Chamber of Commerce and 
project staff appeared to function well. A total of %k commercial 
and Industrial firm* conducted learning activities for the Academy 
students during FY 1 97*; this represents an Increase of 83* over 
FY 1973* Participating employers also served on the Academy Board of 
Directors and were Involved In recruiting both new students and employer 

resources. v 

1 

Employers were grouped Into clusters based upon commonal I ties In 
products and career categories represented. It was Intended that the 
Instructional content and objectives would be Integrated among employers 
within clusters. This was to be accomplished through "cluster meetings 0 
and cooperative program development. To a large extent this did not 
occljr. Since the Integration of activities within clusters Is seen 
as Important to currkulum continuity, It Is recommended that means of 
accomplishing the Intent be pursued by effecting old procedures or 
.designing new ones. 

c 
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The pool of employer resources available for student selection 

during FY \37k was more than sufficient to meet student needs and 

interests in Career Exploration. In some cases inefficient scheduling 

produced problems in matching individual students with learning 

activities, but overall the process went well. The situation was 

different for Career Specialization. Host students who requested these 

activities could not be scheduled for what they wanted. This represents 

a serious shortcoming in program conduct. It Is recommended that exten- 

*> • 
slve effort be applied to developing the resources for Career • 

Specialization, promoting the activities among students, and enabling 

students to be scheduled into the activities of tbeir choice. 

The course plans and student contracts available for employer 
learning activities were generally good, but lacking in specificity. 
Of more basic concern, only about half of those plans which should 
have been produced were actually extant for evaluation. This signals 
poor documentation at best, and may be indicative of more substantial 
problems. It Is recommended that all course plans and student 
contracts be produced for Inspection before the learning activities 
they cover comnfence. The curricula can be evaluated only after this 
condition has been met. 

Available test results Indicated that students learned from their 
Career Development activities. Surveys Indicated that students also 
felt these activities were very worthwhile and well conducted. Stri- 
dent opinions were strongly positive about most aspects of both the 
Career Exploration and Career Specialization courses. 

1 • iai 



Overall, tt can be satd that Career Development resources were 
effectively Identified and recruited. Most students (except In 
Career Specialization) vmr* able to participate In the learning 
activities they wanted. As a result of this participation students 
learned and perceived the employer courses very positively. The 
weak links were the implementation of Career Specialization and Indi- 
vidual program documentation. 

Career Guidance. The Guidance Qroups constituted the major 
instructional activity within this program subdivision. On all 
measures this activity was Identified as a problem area. Tests 
of knowledge administered to a random sample of students suggested 
low efficiency In transmitting Information. A full 75* of the ran- 
dom sample students selected the Guidance Groups as the least 
worthwhile program element. Roughly half of all the students In the 
Academy felt that the guidance groups were not beneficial. Only 
It of the FY 1974 graduates Indicated that the guidance groups were 
the most useful part of the Academy program.. The Needs Research and 
Personal Position Audit was operational I zed as a segment of the 
Guidance Group curriculum fpr tenth and eleventh graders. Documentat l( 
of the conduct of this activity was insufficient to permit evaluation 
In any substantive way. These results clearly indicate that the 
Guidance Groups deserve a major rethinking. Since they have been 
designed as an instructional activity, it ts Important to create 

the motivational climate and instructional content whereby learning 
may occur. 



Transactional Activities were operational I zed as Life Skills 
Specializations, a parallel to Career Specializations with a human 

/ « 

services emphasis. Since fewer than 10 students participated in this 

8 

activity* evaluation could not be conducted in any generaHzeable 

way* This activity has been combined with Career Specialization, and 

thus is of no future concern as a di screte enti ty. 

Student placement was not undertaken as a planned activity of 

Che guidance staff. However, post-secondary pursuits are considered 

to be an important variable affected by the program as a whole. In 

this regard, the outcomes of student placement were very encouraging. - 

In a survey conducted at the end of the summer, 53% of the Academy 

graduates were about to enter post-secondary education. Another 15% 

were employed, 10$ in the military service, 3* homemakers, and \3% 

still planning. A full 71t Indicate^ that they were actively planning 

a career; and 6k% said that the Academy experiences had helped them 

f 

to select a career* About half further indicated that Academy person- 

' r J - 

nel had been of help In getting a job or getting into a training/ 
education program. 

* Overall It must be concluded that the Implementation-of the guid- 
ance program did not proceed well. Students evidenced a predominantly 
negative attitude; documentation was often inadequate; in some respects 
It appeared that' planned activities were not actually implemented. The 
outcomes of student placement, however, were very encouraging. It 
does appear that the guidance element needs reconsideration In form 
and implementation. 



*. Basic Skills . The Basic Skills subdivision was implemented as • 
the Individualized Learning, Center. hv^lvldual Izrftlon was apparently 
effected. Thfe results from a staff checkl 1st supported the proper^ 
use of individual ired material*. Students felt that^the program allowed 
them to progress at their own rate and tjhat the mater lals were provid- 
ing for their Individual needs. A field Investigation also demonstrated 
that hundreds of different learning activities were In active use 
by students. This high level ^of Individual Izatibn should be continued. 

The documentation systems utilized in the Individualized Learning 
Center are necessarily complex in order to accommodate personalized 
student programs. They functioned well providing operational and 
evaluative information. A computer system was designed and partially 

e 

developed to serve both instructional »and research purposes, A good 

deal of devtelopment remains to be done. This effort should be 

supported* to the degree possible as a facilitator for large scale 

• .... 

Implementation. 

Both students and staff had a. favorable opinion of the materials 
utilized In the Individual i zed Learning Center. Student} felt 
that the learning resources were above average and that they were 
learning more than in their previous schools. Staff ^Indicated that- 
the Interest level of the materials could be improved. This might 
alleviate .thfc attendance problems which were noted. Such development 
should be encouraged. 

Overall, the Basic Skills program seemed to function well. 
Students were provided for Individually. The documentation systerrts 
were adequate, but could be Improved If time and resources can be 
applied to computer systems development, particularly In operational 

/ 
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utility. Continuing to develop the Interest value Inherent In 
the materials was also advised. 

! 

■ i 

S ummatlve Lvalautlon \ 

* j 

The summatlve evaluation design was based upon two experimental 
groups and two control groups. The experimental groups were th^ACE 
group (twelfth graders in the program for their second year) and the 
ACE-Olney group (tenth and eleventh graders in the program foV their ( 
first year). The control groups were the Comparison group (studehts 
who had been accepted but dropped out) and the -Context group (students 
who were randomly selected from the sending school). The latter 
student groups were non-equivalent controls since random assignment to 
experimental and controV situations could not be accomplished. The 
ACE group needed to' be anlayzed Independently because It was selected 
from a city-wide population of. public, parochial and private school 
student* who had participated In the program for two years. The 
ACE-Olney group was analyzed relative to the Context and Comparison 
groups because all were drawn from a single Philadelphia high school, 
Oiney. The Comparison group was intended to control for Interest In 
career education since they applied for the program (tuft dropped before 
it begain). However, their group characteristics proved* to be . 
sufficiently different from the Academy students to m#ke the validity 
of comparison dubious. The Context group was Intended to depict the 
"typical" high school student. In this It wcs successful, limited 
only by the self-selection of students who elected to take the 
tests. 4 1 

1 



Both experimental and control students were administered a 
group of summative tests at the beginning and end of the school 

year. These tests consisted of the Comprehensive Tests of Basic * 

< 

Skills (CTBS) to measure traditionally-conceived Cognitive 
development, the Caijeer Maturity Inventory (CMI*) to measure * 
career awareness and competencies, and the Assessment of Student 
Attitudes Scale (ASA) to measure attitudes toward learning environ- 
nients. 

The ^ummative design was thus a quasi -experimental design with 
a pretest and posttest battery of instruments administered to 
experimental groups and non-equi valervt contr61s* This design, Is \ 
limited in its general izeabi 1 ity because ttie degree to whfch exper- 
(mental group results and Control group results may be deemed validly 
comparable is questionable,. The design was, however, judged to be 
reasonable given the early state of project development. Results should 
be broadly indicative of trends, if not precisely interpretable 
as effects .of an experimental educational program. The conclusion 

and recommendations drawn from the s&nmative evaluation of RBS 1 

• - c ' * 
Experience Based Career Education 'program are here presented in terms 

** • # 

of hypotheses formulated. as being central to the intended success of 
J 

the program. 
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Student Effects Hypotheses^ It was found that the Academy 
students gained significantly over the course of the year in 
many basic skills areas measured by the Comprehensive Tests of 
Baste Skills. The grade equivalent gains were as follows: 
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Group 










Subtest ^ — ^ 


ACE 


ACE-Olnev 


Cornea r t son 


Context 


rvctiu 1 1 1 y <j 










Vocabulary 


,3 


.6 


.9 




Comprehension 


.9 . 


-.1 


.7 


.i 


Total 


• 6 


.k 


.8 


.i 


Arithmetic 










Computation 


.5 


.7 


.3 


.* 


Concepts - 


.7 


.9 


-.1 


.6 


Appl teat tons 
Tota' 


1.1 


.7 


-.3 


-.6 


. .7 


.8 


.2 


.2 


Average Gain 


.68 


.59 


.3* N 


r .1* 1 



As can be seen from these results Academy students gained substantially 
more than public school students In nationally normed grade equivalence 
units. These* differences in galrts were not, however, statistically 
significant In the analyses reported herein* This suggests that 
the Academy Is at least as effective -as* the public schools In producing 
growth Irj basic skills. This hypothesis cahnot be definitively tested 
without true controls (random assignment of students). Present 
res tilts with non-equivalent control subjects depict Academy, 



students Insreleatlon to public sctiool students whos? statistical 
comparability fs an unknown. v / * 

' Regarding the career maturity variables as nw/asured J>y the Career 
Maturity Inventorying first year students (ACE/oiney) gained slgnt-, 
flcantly over the course of the year. These gamins were slgnlf Icantjyr 

greater In each case than the growth evidenced by control students, « 

m ( j . f 

This suggests thajf the Academy program, contributes more to the career 
maturity of students than pubMc school programs. Agato, a definitive^ 

test of this would require randomly assigned experimental and cpntr<?1 t 

* * t * 

groups. " ' ' f 

Academy students generally demonstrated a more positive attitude 
toward school than control students as /measured by the Assessment 



of Student Attitudes Scale* This suggests that the Academy 

experience promotes favorable student opinions of their learning ' 

« • 
envj ronmentt This could be an important motivational factor, but 

definitive conclusions must await testing under equivalent control 

conditions. 

The testing of student effects hypotheses generally supported 
Academy student growth in basic ski Us and Career maturity, as well 
as the development of favorable attitudes regardlnq learning ftnvir- 
onments. In some cases the data suggested that the Academy Is more 

... * 4 

effective than traditional school programs In promoting the learning . 
and attitudes represented by the measures employed. However,, all » 
analyses were limited in conclusiveness by the^fact that exper- 
imental and control groyps-rWre non-equi valent. The major recommenda- 
tfon to be derived from this is that future study should include a 
randomized student population design. A second recommendation is that 
the instrumentation should be carefully assessed for appropriateness 

V * 

and intepretabil I ty. During FY l c )'/4 s<»me instruments wore ust»«l with 
a knowledge of their limitations because they were the best jvtiihible. 
To thS extent "poss i ble, Instrument modi I ic«it ion should he \indurt*il««»ri 
to reduce limitations incurred. A third recomuvtuUit ion is rfrrivD.I 
I mm l he observation that in some ru'.es second v<jr students <!»<! jw** 
do lis well ns first year 1 students. ruluri., tlfsiqns should » ml UhU* ' flu 
capability for analyzing multiple year effects. 

* Other Effects Hypotheses . In the investigation of these effects 
it was found that business community resources, could be made available 
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to* meet student needs and interests In, the Academy program. Some 
Inefficiency In the utilization of these resources was noted, but 
program feasibility In term* of attracting community Input was 
established. It Is recommended that an Increased effort be made In 
scheduling to get students Into those learning activities expressed 
as their first choice. A more complete expression of student preference 
should also be elicited and documented. Improvements are needed both 
in the availability of resources and scheduling efficiency for Career 
Special Izat Ion to meet Its object!. *es. 

Regarding employer attitude and commitment to the program, the 
hypothesis stated could not be directly tested.' It was noted, how- 
ever, that employers seemed to be generally satisfied with the program, 
and that fewer than 20% dropped out -during FY 197.4". The major employer 
complaint seemed to be In reference to a lack of feedback from the 
Academy regarding employer program development, implementation and 
effectiveness. It Is recommended that this concern be further Invest- 

0 

gated and attention given td Its remediation. * V 

Parent attitude toward the program was found to.be especially 
positive. >ar<#flts Indicated that the Academy had provided a valuable 
learning and developmental experience for their children. .Further, 

they felt that such results would not have been as obtainable by their 

« 

children In the public schools. Clearly, this constituent body ^ees 

a valuable role for EBCE In public education and their continued support 

and participation should be encouraged. . » 

Progress was also seen relating to another Important constituent 
body: the p'ubl tc schools. Relationships with the Philadelphia Publ ic: 
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Schools have developed from a status of coexistence during the fJrst year 
to a cooperative venture during FY 1974. This seems to be moving 
toward a situation wherein the public schools will sponsor the 
Academy program In the future. Such development approaches the major 

• 

project goal of making the EBCE experience available to many students, 
not just those In demonstration programs. Substantial reductions ' 
in costs this year will undoubtedly also foster progress toward 

o 

this goal by making larger scale adoption more cost --feasible. The 
development of relationships" wi th the Philadelphia PubUc Schools was 
thus seen as very successful, f ' 

The attractiveness of £BCE does not seem to be limited to Phila- 
delphia. The model has generated Interest among public school 
districts throughout the Pennsylvania - Delaware - New Jersey ^rea. 
A mailing of Information about 'the Academy^ resulted Iri positive response 
from close to 10Q. school districts irrterestedNin finding out how 
EBCE might be utilized «in their schools. This was an initial effort 
In testing the market, and it was an encouraging one. It'should * 
be attempted to follow up on the Interest which has already been 
expressed, £rid further dissemination of »£BCE should become an- 
increasing priority. 

Non-Hypothesized Effects .' The conclusion to this section is 

X ■ 

fittingly concerned with FY ,J97*hgraduates of the Academy for 

• '« * > , • ' 

Career Education. Graduates cfemonstrated very positive attitudes 

toward the program. They indicated that their Academy experiences 

had been beneficial in 'planning their careers I, and, as a group 
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they seemed to be taking more positive action in career pursuit 
than cbntrol group graduates. The longer term program effects are 
probably the most Important ones, and they should be accorded priority 
In future evaluation design. 

Final Words 



In the totality of the Final Evaluation Report for FY , 
extensive Information 'about the RBS Career Education Program has been 
presented. In the process important points may have been obscured t¥ 
by the breadth of the ground covered or simple reader fatigue . These 
M flnal words 11 have been reserved to highlight several conclusions, 
assert igns^and issues which are seen as superordtnate in program • 
evaluation. To wit: 



The term "Final Report 11 is hopefully a. misnomer. It Is "final 11 
for FY 197* f but FY 197* stands as the successor to> FY 1973 and the 
predecessor of FY 1975. The project and Its evaluation must be under- 
stood in the perspective of a progressive activity. The summative 
evaluation of program effects reported here does not represent a 

I 

conclusive statement*of effects; it represents* only the best state- 
ment possible at this point In time. The present M best statement 11 Is 
distinguished from a "conclusive statement" by the lack of adequate 
control groups, gaps in instrumentation, and difficulties In 
design Implementation. These discrepancies between the ideal and the 
actual are neither staggering nor shocking; they are factors to be 
confronted and surmounted as the evaluation effort develops. The 
limitations are fewer this 'year than last, and nertt year further ' 

4 * 

Sevelopmen^wil 1 be In evidence. «From this it Is important to lAider- 
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stand that the FY 1974 evaluation represents an Interim stage In 
assessing effects, not a final judgements Last year there were no * 
comparison group data at all and Instrumentation was primitive. 
This year 'there were non-equivalent controls and developmental 
instrumentation. Next year there will be true control subjects 
and more complete instrumentation. The fact that evaluation findings 
have limitations due to thestage of evaluation development does not 
p*Y se detract frdm the significance of the results If they are 
interpreted in cognizance of this inherent limitation. 

Asf should be evident from the scope of the Final Report * the 
bV^dth of topics covered by the evaluation design during FY 1974 
was extensive. The evaluation tasks were both multitudinous and 

e 

diverse. The resources applied to the effdrt werealso substantial 
and deployed with all the administrative efficiency available. The 
extenslveness and multl faceted naturts of the tasks at hand, however, 

• * * _ 

mitigated against the accomplishment of all tasks with the 
deliberating depth' whlch*wpuld be considered optimal. Various 
integrations of findings, Interpretations of outcomes, cross substan- . 

r 

tiatlorrs of results, and complex analyses were seen as deslrealAe 

/ 

but simply could not be done* This may, be interpreted as spreading 

f 

the resources too thinly. It Is a fact of life in comprehensive 

program evaluation on the one hand, but It contains a recommendation 

/ 

on the other.* Greater selectivity In the topics fo* evaluation should 
m 

be exert! zed a; the evaluation effort develops. It U asserted that 
the resultant gains In depth enabled more than off^t the losses 
lp scope incurred. v 
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The hope that an evaluation will result In a simple "good" or 
"bad" rating for the subject of the evaluation Is no more fervent In 
the mind of the reader of evaluation reports than it fs in the mind 
of the evaluator. This fantasy, however, only comes true In public 
relations documents. In an effort to aid the readers of this report, 
and perhaps induce a false sense of security, a brief statement of 
perceived net effects of the Academy program is here presented. The 
principal EBCE constituent groups - students, parents, employers 
and public school representatives - were very positive about the, 
value t)f the program personally and as an educational contribution. 

m 

Students gained substantial ly^in knowledge, personal career awareness 

arid affective dimensions durlnq thel^r experience l,n the oroaram. In 
many cases student development In the program was superior to that 
apparent In the public schools. In* no case was the development of 
public school students superior to that of the students in the program. 

Problems tn the implementation of the model have been Identified which 
suggest the need for continued development and refinement. Real Inter- 
est In this type of program has been expressed By many public school 
districts. The fact that the program can be Implemented with reasonabl 
Indications of accomplishing Intended objectives which are valued 
by major constituent bodies argues for the significance of EBCE 
as a viable educational Innovation. Its potent\a1 should' continue to 
be pursued. 
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